Lab book Fluotracify

1 Technical Notes

1.1 README

1.1.1 General:

  • This file corresponds to my lab book for my doctoral thesis tackling artifact correction in Fluorescence Correlation Spectroscopy (FCS) measurements using Deep Neural Networks. It also contains notes taken during the process of setting up this workflow for reproducible research.
  • This file contains explanations of how things are organized, of the workflow for doing experiments, changes made to the code, and the observed behavior in the "* Data" section.
  • The branching model used is described in this paper. Therefore: if you are interested in the "* Data" section, you have to git clone the data branch of the repository. The main branch is clean from any results, it contains only source code and the analysis.
  • This project is my take on Open-notebook science. The idea was postulated in a blog post in 2006:

    … there is a URL to a laboratory notebook that is freely available and indexed on common search engines. It does not necessarily have to look like a paper notebook but it is essential that all of the information available to the researchers to make their conclusions is equally available to the rest of the world —Jean-Claude Bradley

  • Proposal on how to deal with truly private data (e.g. notes from a confidential meeting with a colleague), which might otherwise be noted in a normal Lab notebook: do not include them here. Only notes relevant to the current project should be taken

1.1.2 Code block languages used in this document

# This is a sh block for shell / bash scripting. In the context of this file,
# these blocks are mainly used for operations on my local computer.
# In the LabBook.html rendering of this document, these blocks will have a
# light green colour (#F0FBE9)
# This block can open and access tmux sessions, used for shell scripting on
# remote computing clusters.
# In the LabBook.html rendering of this document, these blocks will have a
# distinct light green colour (#E1EED8)
# This is a python block. In the context of this file, it is seldomly used
# (only for examplary scripts.)
# In the LabBook.html rendering of this document, these blocks will have a
# light blue colour (#E6EDF4)
# This is a jupyter-python block. The code is sent to a jupyter kernel running
# on a remote high performance computing cluster. Most of my jupyter code is
# executed this way.
# In the LabBook.html rendering of this document, these blocks will have a
# light orange colour (#FAEAE1)
;; This is a emacs-lisp block, the language used to customize Emacs, which is
;; sometimes necessary, since the reproducible workflow of this LabBook is
;; tightly integrated with Emacs and org-mode.
;; In the LabBook.html rendering of this document, these blocks will have a
;; light violet colour (#F7ECFB)
This is a literal example block. It can be used very flexibly - in the context
of this document the output of most code blocks is displayed this way.
In the LabBook.html rendering of this document, these blocks will have a light
yellow colour (#FBFBBF)
This is a literal example block enclosed in a details block. This is useful to
make the page more readable by collapsing large amounts of output.
In the Labbook.html rendering of this document, the details block will have a
light grey colour (#f0f0f0) and a pink color when hovering above it.

1.1.3 Experiments workflow:

  1. Create a new branch from main
  2. Print out the git log from the latest commit and the metadata
  3. Call the analysis scripts, follow the principles outlined in Organization of code
  4. All machine learning runs are saved in data/mlruns, all other data in data/#experiment-name
  5. Add a ** exp-<date>-<name>" section to this file under Data
  6. Commit/push the results of this separate branch
  7. Merge this new branch with the remote data branch

1.1.4 Example for experimental setup procedure

1.1.4.1 Setting starting a jupyter kernel from a remote jupyter session using emacs-jupyter in org babel

1.1.5 tools used (notes)

1.1.5.1 Emacs magit
1.1.5.2 jupyter

1.2 Template for data entry and setup notes:

1.2.1 exp-#date-#title

1.2.1.1 git:
git log -1
1.2.1.2 System Metadata:
import os
import pprint

ramlist = os.popen('free -th').readlines()[-1].split()[1:]

print('No of CPUs in system:', os.cpu_count())
print('No of CPUs the current process can use:',
      len(os.sched_getaffinity(0)))
print('load average:', os.getloadavg())
print('os.uname(): ', os.uname())
print('PID of process:', os.getpid())
print('RAM total: {}, RAM used: {}, RAM free: {}'.format(
    ramlist[0], ramlist[1], ramlist[2]))

!echo the current directory: $PWD
!echo My disk usage:
!df -h
if _long:
    %conda list
    pprint.pprint(dict(os.environ), sort_dicts=False)

1.2.1.3 Tmux setup and scripts
rm ~/.tmux-local-socket-remote-machine
REMOTE_SOCKET=$(ssh ara 'tmux ls -F "#{socket_path}"' | head -1)
echo $REMOTE_SOCKET
ssh ara -tfN \
    -L ~/.tmux-local-socket-remote-machine:$REMOTE_SOCKET
rm: cannot remove 'home/lex.tmux-local-socket-remote-machine': No such file or directory
ye53nis@ara-login01.rz.uni-jena.de's password:              
/tmp/tmux-67339/default                
> ye53nis@ara-login01.rz.uni-jena.de's password:            
1.2.1.4 SSH tunneling

Different applications can be run on the remote compute node. If I want to access them at the local machine, and open them with the browser, I use this tunneling script.

ssh -t -t ara -L $port:localhost:$port ssh $node -L $port:Localhost:$port

Apps I use that way:

  • Jupyter lab for running Python 3-Kernels
  • TensorBoard
  • Mlflow ui
1.2.1.5 jupyter scripts

Starting a jupyter instance on a server where the necessary libraries are installed is easy using this script:

conda activate tf
export PORT=9999
export XDG_RUNTIME_DIR=''
export XDG_RUNTIME_DIR=""
jupyter lab --no-browser --port=$PORT

On the compute node of the HPC, the users' environment is managed through module files using the system Lmod. The export XDG_RUNTIME_DIR statements are needed because of a jupyter bug which did not let it start. Right now, ob-tmux does not support a :var header like normal org-babel does. So the $port variable has to be set here in the template.

Now this port has to be tunnelled on our local computer (See SSH tunneling). While the tmux session above keeps running, no matter if Emacs is running or not, this following ssh tunnel needs to be active locally to connect to the notebook. If you close Emacs, it would need to be reestablished

1.2.2 Setup notes

1.2.2.1 Setting up a tmux connection from using ob-tmux in org-babel
  • prerequisite: tmux versions need to be the same locally and on the server. Let's verify that now.
    • the local tmux version:
      tmux -V
      
      tmux 3.0a
      
    • the remote tmux version:
      ssh ara tmux -V
      
      ye53nis@ara-login01.rz.uni-jena.de's password:
      tmux 3.0a
  • as is described in the ob-tmux readme, the following code snippet creates a socket on the remote machine and forwards this socket to the local machine (note that socket_path was introduced in tmux version 2.2)
    REMOTE_SOCKET=$(ssh ara 'tmux ls -F "#{socket_path}"' | head -1)
    echo $REMOTE_SOCKET
    ssh ara -tfN \
        -L ~/.tmux-local-socket-remote-machine:$REMOTE_SOCKET
    
    ye53nis@ara-login01.rz.uni-jena.de's password:  
    /tmp/tmux-67339/default    
    > ye53nis@ara-login01.rz.uni-jena.de's password:
  • now a new tmux session with name ob-NAME is created when using a code block which looks like this: #+BEGIN_SRC tmux :socket ~/.tmux-local-socket-remote-machine :session NAME
  • Commands can be sent now to the remote tmux session, BUT note that the output is not printed yet
  • there is a workaround for getting output back to our LabBook.org: A script which allows to print the output from the tmux session in an #+begin_example-Block below the tmux block by pressing C-c C-o or C-c C-v C-o when the pointer is inside the tmux block.
1.2.2.2 emacs-jupyter Setup

Emacs-jupyter aims to be an API for a lot of functionalities of the jupyter project. The documentation can be found on GitHub.

  1. For the whole document: connect to a running jupyter instance
    1. M-x jupyter-server-list-kernels
      1. set server URL, e.g. http://localhost:8889
      2. set websocket URL, e.g. http://localhost:8889
    2. two possibilities
      1. kernel already exists \(\to\) list of kernels and kernel-ID is displayed
      2. kernel does not exist \(\to\) prompt asks if you want to start one \(\to\) yes \(\to\) type kernel you want to start, e.g. Python 3
  2. In the subtree where you want to use jupyter-python blocks with org babel
    1. set the :header-args:jupyter-python :session /jpy:localhost#kernel:8889-ID
    2. customize the output folder using the following org-mode variable:
      (setq org-babel-jupyter-resource-directory "./data/exp-test/plots")
      
      ./data/exp-test/plots
      
  3. For each individual block, the following customizations might be useful
    1. jupyter kernels can return multiple kinds of rich output (images, html, …) or scalar data (plain text, numbers, lists, …). To force a plain output, use :results scalar. To show the output in the minibuffer only, use :results silent
    2. to change the priority of different rich outputs, use :display header argument, e.g. :display text/plain text/html prioritizes plain text over html. All supported mimetypes in default order:
      1. text/org
      2. image/svg+xml, image/jpeg, image/png
      3. text/html
      4. text/markdown
      5. text/latex
      6. text/plain
    3. We can set jupyter to output pandas DataFrames as org tables automatically using the source block header argument :pandoc t
    4. useful keybindings
      • M-i to open the documentation for wherever your pointer is (like pressing Shift-TAB in Jupyter notebooks)
      • C-c C-i to interrupt the kernel, C-c C-r to restart the kernel

1.2.3 Notes on archiving

1.2.3.1 Exporting the LabBook.org to html in a twbs style
  • I am partial to the twitter bootstrap theme of html, since I like it's simple design, but clear structure with a nice table of contents at the side → the following org mode extension supports a seemless export to twitter bootstrap html: https://github.com/marsmining/ox-twbs
  • when installed, the export can be triggered via the command (org-twbs-export-as-html) or via the keyboard shortcut for export C-c C-e followed by w for Twitter bootstrap and h for saving the .html
  • Things to configure:
    • in general, there are multiple export options: https://orgmode.org/manual/Export-Settings.html
    • E.g. I set 2 #+OPTIONS keywords at the begin of the file: toc:4 and H:4 which make sure that in my export my sidebar table of contents will show numbered headings till a depth of 4.
    • I configured my code blocks so that they will not be evaluated when exporting (I would recommend this especially if you only export for archiving) and that both the code block and the output will be exported with the keyword: #+PROPERTY: header-args :eval never-export :exports both
    • To discriminate between code blocks for different languages I gave each of them a distinct colour using #+HTML_HEAD_EXTRA: <style... (see above)
    • I had to configure a style for table, so that the
      • display: block; overflow-x: auto; gets the table to be restricted to the width of the text and if it is larger, activates scrolling
      • white-space: nowrap; makes it that there is no wrap in a column, so it might be broader, but better readable if you have scrolling anyway
  • Things to do before exporting / Troubleshooting while exporting:
    • when using a dark theme for you emacs, the export of the code blocks might show some ugly dark backgrounds from the theme. If this becomes an issue, change to a light theme for the export with M-x (load-theme) and choose solarized-light
    • only in the data branch you set the git tags after merging. If you want to show them here, execute the corresponding function in Git TAGs
    • make sure your file links work properly! I recommend referencing your files relatively (e.g. [ [ f ile:./data/exp-XXXXXX-test/test.png]] without spaces). Otherwise there will be errors in your Messages buffer
    • There might be errors with your code blocks
      • e.g. the export function expects you to assign a default variable to your functions
      • if you call a function via the #+CALL mechanism, it wants you to include two parentheses for the function, e.g. #+CALL: test()
    • check indentation of code blocks inside lists
    • add a details block around large output cells. This makes them expandable. I added some #+HTML_HEAD_EXTRA: <style... inspired by alhassy. That's how the details block looks like:
      #+begin_details
      
      #+end_details
      
    • If you reference a parameter with an underscore in the name, use the org markdown tricks to style them like code (== or ~~), otherwise the part after the underscore will be rendered like a subscript: under_score vs underscore
  • Things to do after exporting:
    • In my workflow, the exported LabBook.html with the overview of all experiments is in the data folder. If you move the file, you will have to fix the file links for the new location, e.g. via "Find and replace" M-%:
      • if you move the org file → in the org file find [[file:./data/ and replace with [[file:./ → then export with C-c C-e w h
      • if you export first with C-c C-e w h and move the html file to data → in the html file find ./data and replace with .

1.3 Organization of git

1.3.1 remote/origin/main branch:

  • contains all the source code in folder src/ which is used for experiments.
  • contains the LabBook.org template
  • contains setup- and metadata files such as MLproject or conda.yaml
  • the log contains only lasting alterations on the folders and files mentioned above, which are e.g. used for conducting experiments or which introduce new features. Day-to-day changes in code

1.3.2 remote/origin/exp### branches:

  • if an experiment is done, the code and templates will be branched out from main in an #experiment-name branch, ### meaning some meaningful descriptor.
  • all data generated during the experiment (e.g. .csv files, plots, images, etc), is stored in a folder with the name data/#experiment-name, except machine learning-specific data and metadata from `mlflow` runs, which are saved under data/mlruns (this allows easily comparing machine learning runs with different experimental settings)
  • The LabBook.org file is essential
    • If possible, all code is executed from inside this file (meaning analysis scripts or calling the code from the scr/ directory).
    • All other steps taken during an experiment are noted down, as well as conclusions or my thought process while conducting the experiment
    • Provenance data, such as metadata about the environment the code was executed in, the command line output of the code, and some plots

1.3.3 remote/origin/develop branch:

  • this is the branch I use for day to day work on features and exploration. All of my current activity can be followed here.

1.3.4 remote/origin/data branch:

  • contains a full cronicle of the whole research process
  • all #experiment-name branches are merged here. Afterwards the original branch is deleted and on the data branch there is a Git tag which shows the merge commit to make accessing single experiments easy.
  • the develop branch is merged here as well.

1.3.5 Git TAGs

1.3.5.1 Stable versions:
1.3.5.2 All tags from git:
git push origin --tags
git tag -n1
exp-200402-test Merge branch 'exp-200402-test' into data
exp-200520-unet Merge branch 'exp-310520-unet' into data
exp-200531-unet Merge branch 'heads/exp-310520-unet' into data
exp-201231-clustsim exp-201231-clustsim
exp-210204-unet Add exp-210204-unet LabBook part 3
exp-310520-unet move exp-310520-unet to data branch manually

1.4 Organization of code

1.4.1 scripts:

1.4.2 src/

1.4.2.1 fluotracify/
  1. imports/
  2. simulations/
  3. training/
  4. applications/
  5. doc/
1.4.2.2 nanosimpy/
  • cloned from dwaithe with refactoring for Python 3-compatibility

1.5 Changes in this repository (without "* Data" in this file)

1.5.1 Changes in LabBook.org (without "* Data")

1.5.1.1 2022-02-19
  • Add #+HTML_HEAD_EXTRA: <style... for table to enable scrolling if the table overflows
1.5.1.2 2021-12-16
  • Add details blocks, corresponding #+HTML_HEAD_EXTRA: <style... and documentation in Notes on archiving
1.5.1.3 2021-08-05
  • Rename master branch to main branch
1.5.1.4 2021-04-04
  • Add #+OPTIONS: H:4 and #+OPTIONS: toc:4 to show up to 4 levels of depth in the html (twbs) export of this LabBook in the table of contents at the side
  • I added Notes on archiving
1.5.1.5 2020-11-04
1.5.1.6 2020-05-31
  • extend general documentation in README
  • Add code block examples
  • extend documentation on experiment workflow
  • move setup notes from README to "Template for data entry and setup notes"
  • remove emacs-lisp code for custom tmux block functions (not relevant enough)
  • change named "jpt-tmux" from starting a jupyter notebook to starting jupyter lab. Load a conda environment instead of using Lmod's module load
1.5.1.7 2020-05-07
  • extend documentation on git model
  • extend documentation on jupyter setup
1.5.1.8 2020-04-22
  • added parts of README which describe the experimental process
  • added templates for system metadata, tmux, jupyter setup
  • added organization of code
1.5.1.9 2020-03-30
  • set up lab book and form git repo accoring to setup by Luka Stanisic et al

1.5.2 Changes in src/fluotracify

2 Data

2.1 exp-310520-unet

2.1.1 Connect

2.1.1.1 Tmux on Ara
ye53nis@ara-login01.rz.uni-jena.de's password:  
/tmp/tmux-67339/default    
> ye53nis@ara-login01.rz.uni-jena.de's password:

Test:

pwd
(tensorflow_nightly) [ye53nis@login01 drmed-git]$ pwd
/beegfs/ye53nis/drmed-git
pwd
2.1.1.2 Compute node for script execution
srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
(base) [ye53nis@node151 drmed-git]$
2.1.1.3 Jupyter on Ara
  1. Request compute node via tmux
    srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
    
    (base) [ye53nis@node189 drmed-git]$
    
    cd /home/ye53nis/DOKTOR
    
  1. Start Jupyter Lab
    [I 10:54:50.672 LabApp] JupyterLab extension loaded from /home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/jupyterlab
    [I 10:54:50.673 LabApp] JupyterLab application directory is /home/ye53nis/.conda/envs/tensorflow_nightly/share/jupyter/lab
    [I 10:54:50.678 LabApp] Serving notebooks from local directory: /home/ye53nis/DOKTOR
    [I 10:54:50.678 LabApp] The Jupyter Notebook is running at:
    [I 10:54:50.678 LabApp] http://localhost:8889/?token=b210da5d453ac75f8f246e3c23917c94578c516ecfe0d95b
    [I 10:54:50.678 LabApp]  or http://127.0.0.1:8889/?token=b210da5d453ac75f8f246e3c23917c94578c516ecfe0d95b
    [I 10:54:50.678 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
    [C 10:54:50.724 LabApp]
    
        To access the notebook, open this file in a browser:
            file:///home/ye53nis/.local/share/jupyter/runtime/nbserver-365563-open.html
        Or copy and paste one of these URLs:
            http://localhost:8889/?token=b210da5d453ac75f8f246e3c23917c94578c516ecfe0d95b
         or http://127.0.0.1:8889/?token=b210da5d453ac75f8f246e3c23917c94578c516ecfe0d95b
    
sh-5.0$ ye53nis@ara-login01.rz.uni-jena.de's password:              
ye53nis@node020's password: channel 3: open failed: connect failed: Connection refused
channel 3: open failed: connect failed: Connection refused    
Last login: Sun Jul 5 09:52:47 2020 from login01.ara  

I started a Python3 kernel using jupyter-server-list-kernels. Then I added the kernel ID to the :PROPERTIES: drawer of this (and following) subtrees.

python3           7fe3d66b-d802-4f0f-b105-9536a917b816   a few seconds ago    starting   0

Test:

No of CPUs in system: 72
No of CPUs the current process can use: 24
load average: (1.06, 1.01, 0.97)
os.uname():  posix.uname_result(sysname='Linux', nodename='node218', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
PID of process: 10254
RAM total: 199G, RAM used: 6.0G, RAM free: 181G
the current directory: /home/ye53nis/DOKTOR
My disk usage:
Filesystem           Size  Used Avail Use% Mounted on
/dev/sda1             50G  3.2G   47G   7% /
devtmpfs              94G     0   94G   0% /dev
tmpfs                 94G  725M   94G   1% /dev/shm
tmpfs                 94G   75M   94G   1% /run
tmpfs                 94G     0   94G   0% /sys/fs/cgroup
nfs02-ib:/data01      88T   61T   27T  70% /data01
nfs03-ib:/pool/work  100T   79T   22T  79% /nfsdata
nfs01-ib:/home        80T   61T   20T  76% /home
nfs01-ib:/cluster    2.0T  316G  1.7T  16% /cluster
/dev/sda3            6.0G  412M  5.6G   7% /var
/dev/sda6            169G  2.8G  166G   2% /local
/dev/sda5            2.0G   34M  2.0G   2% /tmp
beegfs_nodev         524T  453T   72T  87% /beegfs
tmpfs                 19G     0   19G   0% /run/user/67339
/bin/sh: conda: command not found
{'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
 'SLURM_NODELIST': 'node218',
 'SLURM_JOB_NAME': 'bash',
 'XDG_SESSION_ID': '8541',
 'SLURMD_NODENAME': 'node218',
 'SLURM_TOPOLOGY_ADDR': 'node218',
 'SLURM_NTASKS_PER_NODE': '24',
 'HOSTNAME': 'login01',
 'SLURM_PRIO_PROCESS': '0',
 'SLURM_SRUN_COMM_PORT': '45911',
 'SHELL': '/bin/bash',
 'TERM': 'xterm-color',
 'SLURM_JOB_QOS': 'qstand',
 'SLURM_PTY_WIN_ROW': '52',
 'HISTSIZE': '1000',
 'TMPDIR': '/tmp',
 'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
 'SSH_CLIENT': '10.231.190.186 48592 22',
 'CONDA_SHLVL': '2',
 'CONDA_PROMPT_MODIFIER': '(tensorflow_nightly) ',
 'GSETTINGS_SCHEMA_DIR_CONDA_BACKUP': '',
 'WINDOWID': '0',
 'OLDPWD': '/beegfs/ye53nis/drmed-git',
 'QTDIR': '/usr/lib64/qt-3.3',
 'QTINC': '/usr/lib64/qt-3.3/include',
 'SSH_TTY': '/dev/pts/57',
 'QT_GRAPHICSSYSTEM_CHECKED': '1',
 'SLURM_NNODES': '1',
 'USER': 'ye53nis',
 'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
 'CONDA_EXE': '/cluster/miniconda3/bin/conda',
 'SLURM_STEP_NUM_NODES': '1',
 'SLURM_JOBID': '392142',
 'SRUN_DEBUG': '3',
 'SLURM_NTASKS': '24',
 'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
 'SLURM_STEP_ID': '0',
 'TMUX': '/tmp/tmux-67339/default,47311,2',
 '_CE_CONDA': '',
 'CONDA_PREFIX_1': '/cluster/miniconda3',
 'SLURM_STEP_LAUNCHER_PORT': '45911',
 'SLURM_TASKS_PER_NODE': '24',
 'MAIL': '/var/spool/mail/ye53nis',
 'PATH': '/home/ye53nis/.conda/envs/tensorflow_nightly/bin:/home/lex/Programme/miniconda3/envs/tensorflow_env/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/home/lex/Programme/miniconda3/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
 'GSETTINGS_SCHEMA_DIR': '/home/ye53nis/.conda/envs/tensorflow_nightly/share/glib-2.0/schemas',
 'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
 'SLURM_JOB_ID': '392142',
 'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tensorflow_nightly',
 'SLURM_JOB_USER': 'ye53nis',
 'SLURM_STEPID': '0',
 'PWD': '/home/ye53nis/DOKTOR',
 'SLURM_SRUN_COMM_HOST': '192.168.192.5',
 'LANG': 'en_US.UTF-8',
 'SLURM_PTY_WIN_COL': '206',
 'SLURM_UMASK': '0022',
 'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
 'SLURM_JOB_UID': '67339',
 'LOADEDMODULES': '',
 'SLURM_NODEID': '0',
 'TMUX_PANE': '%2',
 'SLURM_SUBMIT_DIR': '/beegfs/ye53nis/drmed-git',
 'SLURM_TASK_PID': '696',
 'SLURM_NPROCS': '24',
 'SLURM_CPUS_ON_NODE': '24',
 'SLURM_DISTRIBUTION': 'block',
 'https_proxy': 'https://internet4nzm.rz.uni-jena.de:3128',
 'SLURM_PROCID': '0',
 'HISTCONTROL': 'ignoredups',
 '_CE_M': '',
 'SLURM_JOB_NODELIST': 'node218',
 'SLURM_PTY_PORT': '37811',
 'HOME': '/home/ye53nis',
 'SHLVL': '3',
 'SLURM_LOCALID': '0',
 'SLURM_JOB_GID': '13280',
 'SLURM_JOB_CPUS_PER_NODE': '24',
 'SLURM_CLUSTER_NAME': 'hpc',
 'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
 'SLURM_SUBMIT_HOST': 'login01',
 'SLURM_JOB_PARTITION': 's_standard',
 'MATHEMATICA_HOME': '/cluster/apps/mathematica/11.3',
 'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
 'LOGNAME': 'ye53nis',
 'SLURM_STEP_NUM_TASKS': '24',
 'QTLIB': '/usr/lib64/qt-3.3/lib',
 'SLURM_JOB_ACCOUNT': 'iaob',
 'SLURM_JOB_NUM_NODES': '1',
 'MODULESHOME': '/usr/share/Modules',
 'CONDA_DEFAULT_ENV': 'tensorflow_nightly',
 'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
 'SLURM_STEP_TASKS_PER_NODE': '24',
 'PORT': '8889',
 'SLURM_STEP_NODELIST': 'node218',
 'DISPLAY': ':0',
 'XDG_RUNTIME_DIR': '',
 'XAUTHORITY': '/tmp/xauth-1000-_0',
 'BASH_FUNC_module()': '() {  eval `/usr/bin/modulecmd bash $*`\n}',
 '_': '/home/ye53nis/.conda/envs/tensorflow_nightly/bin/jupyter',
 'KERNEL_LAUNCH_TIMEOUT': '40',
 'JPY_PARENT_PID': '7566',
 'CLICOLOR': '1',
 'PAGER': 'cat',
 'GIT_PAGER': 'cat',
 'MPLBACKEND': 'module://ipykernel.pylab.backend_inline'}
2.1.1.4 Tensorboard tunnel, Mlflow ui tunnel
sh-5.0$ ye53nis@ara-login01.rz.uni-jena.de's password:            
ye53nis@node171's password:              
Last login: Sun May 31 15:29:02 2020 from login01.ara

2.1.2 git exp 1

git status
git log -1
(tensorflow_nightly) [ye53nis@node171 drmed-git]$ git status
# On branch exp-310520-unet
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       data/
#       experiment_params.csv
#       mlruns/
#       tramp.YDPCnB
nothing added to commit but untracked files present (use "git add" to track)

(tensorflow_nightly) [ye53nis@node171 drmed-git]$ git log -1
commit 2225d6fa18cca9044960b6e86a56ec9fb4362d5d
Author: Apoplex <oligolex@vivaldi.net>
Date:   Sun May 31 21:34:12 2020 +0200

    Change learning rate

2.1.3 mlflow environment variables

conda activate tensorflow_nightly
cd /beegfs/ye53nis/drmed-git
export MLFLOW_EXPERIMENT_NAME=exp-310520-unet
export MLFLOW_TRACKING_URI=file:./data/mlruns
mkdir data/exp-310520-unet
(tensorflow_nightly) [ye53nis@node171 drmed-git]$

2.1.4 Learning rate schedule

learning_rate = 0.2
        if epoch > 10:
            learning_rate = 0.02
        if epoch > 20:
            learning_rate = 0.01
        if epoch > 40:
            learning_rate = 0.001
        if epoch > 60:
            learning_rate = 0.0001
        if epoch > 80:
            learning_rate = 0.00001

2.1.5 test runs

2.1.5.1 no 1 - experiment creation failed
mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=2 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Mar2020 -P steps_per_epoch=400 -P validation_steps=200
(tensorflow_nightly) [ye53nis@node171 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=2 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Mar2020 -P
   steps_per_epoch=400 -P validation_steps=200
  WARNING:root:Malformed experiment '1'. Detailed error Yaml file './data/mlruns/1/meta.yaml' does not exist.
  Traceback (most recent call last):
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 197, in list_experiments
      experiment = self._get_experiment(exp_id, view_type)
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 260, in _get_experiment
      meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 167, in read_yaml
      raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
  mlflow.exceptions.MissingConfigException: Yaml file './data/mlruns/1/meta.yaml' does not exist.
  INFO: 'exp-310520-unet' does not exist. Creating a new experiment
  WARNING:root:Malformed experiment '1'. Detailed error Yaml file './data/mlruns/1/meta.yaml' does not exist.
  Traceback (most recent call last):
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 197, in list_experiments
      experiment = self._get_experiment(exp_id, view_type)
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 260, in _get_experiment
      meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 167, in read_yaml
      raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
  mlflow.exceptions.MissingConfigException: Yaml file './data/mlruns/1/meta.yaml' does not exist.
  WARNING:root:Malformed experiment '1'. Detailed error Yaml file './data/mlruns/1/meta.yaml' does not exist.
  Traceback (most recent call last):
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 197, in list_experiments
      experiment = self._get_experiment(exp_id, view_type)
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 260, in _get_experiment
      meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
    File "/home/ye53nis/.conda/envs/tensorflow_nightly/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 167, in read_yaml
      raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
  mlflow.exceptions.MissingConfigException: Yaml file './data/mlruns/1/meta.yaml' does not exist.
  2020/05/31 14:00:34 INFO mlflow.projects: === Created directory /tmp/tmpfhabm3p4 for downloading remote URIs passed to arguments of type 'path' ===
  2020/05/31 14:00:34 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/f
  luotracify/training/train.py /beegfs/ye53nis/drmed-git/src 5 0.2 16384 None 2 /beegfs/ye53nis/saves/firstartifact_Mar2020 400 200' in run with ID 'b37371694a004638a6cd7a94d4d2e77f' ===
  2020-05-31 14:00:38.667018: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No
   such file or directory
  2020-05-31 14:00:38.667107: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
  /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favo
  ur of importlib; see the module's documentation for alternative uses
    import imp
  2.3.0-dev20200527
  2020-05-31 14:00:42.422965: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file
   or directory
  2020-05-31 14:00:42.422998: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
  2020-05-31 14:00:42.423020: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node171): /proc/driver/nvidia/version does not exist
  GPUs:  []
  train 0 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set027.csv
  train 1 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set087.csv
  train 2 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set003.csv
  train 3 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set056.csv
  train 4 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set076.csv
  train 5 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set094.csv
  train 6 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set017.csv
  train 7 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set074.csv
  train 8 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set055.csv
  train 9 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set096.csv
  train 10 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set054.csv
  train 11 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set093.csv
  train 12 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set079.csv
  train 13 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set014.csv
  train 14 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set008.csv
  train 15 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set031.csv
  train 16 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set023.csv
  train 17 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set025.csv
  train 18 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set034.csv
  train 19 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set009.csv
  train 20 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set044.csv
  train 21 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set063.csv
  train 22 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set004.csv
  train 23 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set072.csv
  train 24 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set046.csv
  train 25 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set049.csv
  train 26 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set007.csv
  train 27 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set100.csv
  train 28 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set083.csv
  train 29 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set077.csv
  train 30 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set061.csv
  train 31 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set081.csv
  train 32 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set091.csv
  train 33 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set069.csv
  train 34 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set052.csv
  train 35 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set028.csv
  train 36 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set019.csv
  train 37 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set057.csv
  train 38 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set064.csv
  train 39 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set075.csv
  train 40 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set002.csv
  train 41 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set062.csv
  train 42 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set043.csv
  train 43 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set042.csv
  train 44 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set005.csv
  train 45 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set016.csv
  train 46 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set018.csv
  train 47 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set041.csv
  train 48 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set039.csv
  train 49 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set006.csv
  train 50 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set092.csv
  train 51 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set060.csv
  train 52 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set001.csv
  train 53 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set035.csv
  train 54 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set029.csv
  train 55 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set051.csv
  train 56 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set012.csv
  train 57 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set036.csv
  train 58 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set024.csv
  train 59 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set053.csv
  train 60 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set011.csv
  train 61 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set032.csv
  train 62 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set067.csv
  train 63 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set058.csv
  train 64 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set080.csv
  train 65 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set086.csv
  train 66 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set033.csv
  train 67 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set085.csv
  train 68 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set015.csv
  train 69 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set090.csv
  train 70 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set020.csv
  train 71 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set030.csv
  train 72 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set050.csv
  train 73 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set098.csv
  train 74 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set099.csv
  train 75 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set070.csv
  train 76 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set021.csv
  train 77 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set095.csv
  train 78 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set073.csv
  train 79 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set078.csv
  test 80 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set026.csv
  test 81 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set038.csv
  test 82 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set082.csv
  test 83 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set047.csv
  test 84 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set040.csv
  test 85 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set066.csv
  test 86 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set059.csv
  test 87 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set013.csv
  test 88 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set089.csv
  test 89 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set071.csv
  test 90 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set088.csv
  test 91 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set037.csv
  test 92 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set022.csv
  test 93 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set084.csv
  test 94 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set010.csv
  test 95 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set097.csv
  test 96 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set068.csv
  test 97 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set065.csv
  test 98 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set048.csv
  test 99 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set045.csv
  shapes of feature dataframe: (16384, 8000) and label dataframe: (16384, 8000)
  shapes of feature dataframe: (16384, 2000) and label dataframe: (16384, 2000)

  for each 20,000 timestap trace there are the following numbers of corrupted timesteps:
   label001_1    16384
  label001_1    16384
  label001_1    16384
  label001_1    16384
  label001_1    16384
  dtype: int64
  2020-05-31 14:05:15.534482: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical opera
  tions:  AVX2 AVX512F FMA
  To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
  2020-05-31 14:05:15.544530: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz
  2020-05-31 14:05:15.545867: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557b10ed44b0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
  2020-05-31 14:05:15.545896: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
  number of training examples: 6400, number of validation examples: 1600

  ------------------------
  number of test examples: 2000

  input - shape:   (None, 16384, 1)
  output - shape:  (None, 16384, 1)
  2020-05-31 14:05:22.334555: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
  /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the A
  BCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
    if not isinstance(wrapped_dict, collections.Mapping):
  /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Pytho
  n 3.0, use inspect.signature() or inspect.getfullargspec()
    all_param_names, _, _, all_default_values = inspect.getargspec(fn)  # pylint: disable=W1505
  Epoch 1/2
  /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from
  'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
    if not isinstance(values, collections.Sequence):
    1/400 [..............................] - ETA: 0s - loss: 0.7153 - tp: 66972.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 14948.0000 - precision: 1.0000 - recall: 0.8175 - accuracy: 0.8175 - auc: 0.0000e+0
  02020-05-31 14:05:34.823955: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
  WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager
  .profiler) is deprecated and will be removed after 2020-07-01.
  Instructions for updating:
  use `tf.profiler.experimental.stop` instead.
  2020-05-31 14:05:36.334154: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_14_05_36
  2020-05-31 14:05:36.348656: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.trace.json.gz
  2020-05-31 14:05:36.375326: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_14_05_36
  2020-05-31 14:05:36.375452: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.memory
  _profile.json.gz
  2020-05-31 14:05:36.377489: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_14_05_36Dumped tool data for xplane.pb to /tmp/tb/trai
  n/plugins/profile/2020_05_31_14_05_36/node171.xplane.pb
  Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.overview_page.pb
  Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.input_pipeline.pb
  Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.tensorflow_stats.pb
  Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_14_05_36/node171.kernel_stats.pb

  400/400 [==============================] - 651s 2s/step - loss: 0.0019 - tp: 32753052.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 14948.0000 - precision: 1.0000 - recall: 0.9995 - accuracy: 0.9995 - auc: 0
  .0000e+00 - val_loss: 7.3428e-23 - val_tp: 16384000.0000 - val_fp: 0.0000e+00 - val_tn: 0.0000e+00 - val_fn: 0.0000e+00 - val_precision: 1.0000 - val_recall: 1.0000 - val_accuracy: 1.0000 - val_auc: 0.0000e
  +00
  Epoch 2/2
  400/400 [==============================] - 645s 2s/step - loss: 2.1658e-09 - tp: 32768000.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 0.0000e+00 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - au
  c: 0.0000e+00 - val_loss: 8.7165e-23 - val_tp: 16384000.0000 - val_fp: 0.0000e+00 - val_tn: 0.0000e+00 - val_fn: 0.0000e+00 - val_precision: 1.0000 - val_recall: 1.0000 - val_accuracy: 1.0000 - val_auc: 0.0
  000e+00
  400/400 [==============================] - 119s 298ms/step - loss: 9.0078e-23 - tp: 32768000.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 0.0000e+00 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 -
   auc: 0.0000e+00
  2020/05/31 14:29:15 INFO mlflow.projects: === Run (ID 'b37371694a004638a6cd7a94d4d2e77f') succeeded ===
  (tensorflow_nightly) [ye53nis@node171 drmed-git]$ Regard
mlflow ui
(tensorflow_nightly) [ye53nis@node171 drmed-git]$ mlflow ui
[2020-05-31 15:27:54 +0200] [200865] [INFO] Starting gunicorn 20.0.4
[2020-05-31 15:27:54 +0200] [200865] [INFO] Listening at: http://127.0.0.1:5000 (200865)
[2020-05-31 15:27:54 +0200] [200865] [INFO] Using worker: sync
[2020-05-31 15:27:54 +0200] [200871] [INFO] Booting worker with pid: 200871
2.1.5.2 no 2
mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=2 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Mar2020 -P steps_per_epoch=10 -P validation_steps=10
(tensorflow_nightly) [ye53nis@node171 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=2 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Mar2020 -P
 steps_per_epoch=10 -P validation_steps=10
INFO: 'exp-310520-unet' does not exist. Creating a new experiment
2020/05/31 15:45:30 INFO mlflow.projects: === Created directory /tmp/tmp2vv5kiic for downloading remote URIs passed to arguments of type 'path' ===
2020/05/31 15:45:30 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/f
luotracify/training/train.py /beegfs/ye53nis/drmed-git/src 5 0.2 16384 None 2 /beegfs/ye53nis/saves/firstartifact_Mar2020 10 10' in run with ID '7f23f6ba7a244914b3cbbebd731d50a1' ===
2020-05-31 15:45:31.886019: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No
 such file or directory
2020-05-31 15:45:31.886068: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favo
ur of importlib; see the module's documentation for alternative uses
  import imp
2.3.0-dev20200527
2020-05-31 15:45:35.151500: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file
 or directory
2020-05-31 15:45:35.151549: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-05-31 15:45:35.151589: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node171): /proc/driver/nvidia/version does not exist
GPUs:  []
train 0 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set027.csv
train 1 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set087.csv
train 2 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set003.csv
train 3 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set056.csv
train 4 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set076.csv
train 5 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set094.csv
train 6 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set017.csv
train 7 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set074.csv
train 8 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set055.csv
train 9 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set096.csv
train 10 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set054.csv
train 11 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set093.csv
train 12 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set079.csv
train 13 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set014.csv
train 14 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set008.csv
train 15 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set031.csv
train 16 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set023.csv
train 17 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set025.csv
train 18 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set034.csv
train 19 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set009.csv
train 20 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set044.csv
train 21 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set063.csv
train 22 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set004.csv
train 23 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set072.csv
train 24 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set046.csv
train 25 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set049.csv
train 26 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set007.csv
train 27 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set100.csv
train 28 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set083.csv
train 29 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set077.csv
train 30 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set061.csv
train 31 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set081.csv
train 32 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set091.csv
train 33 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set069.csv
train 34 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set052.csv
train 35 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set028.csv
train 36 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set019.csv
train 37 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set057.csv
train 38 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set064.csv
train 39 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set075.csv
train 40 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set002.csv
train 41 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set062.csv
train 42 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set043.csv
train 43 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set042.csv
train 44 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set005.csv
train 45 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set016.csv
train 46 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set018.csv
train 47 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set041.csv
train 48 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set039.csv
train 49 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set006.csv
train 50 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set092.csv
train 51 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set060.csv
train 52 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set001.csv
train 53 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set035.csv
train 54 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set029.csv
train 55 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set051.csv
train 56 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set012.csv
train 57 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set036.csv
train 58 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set024.csv
train 59 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set053.csv
train 60 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set011.csv
train 61 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set032.csv
train 62 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set067.csv
train 63 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set058.csv
train 64 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set080.csv
train 65 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set086.csv
train 66 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set033.csv
train 67 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set085.csv
train 68 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set015.csv
train 69 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set090.csv
train 70 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set020.csv
train 71 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set030.csv
train 72 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set050.csv
train 73 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set098.csv
train 74 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set099.csv
train 75 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set070.csv
train 76 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set021.csv
train 77 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set095.csv
train 78 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set073.csv
train 79 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set078.csv
test 80 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set026.csv
test 81 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set038.csv
test 82 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set082.csv
test 83 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set047.csv
test 84 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set040.csv
test 85 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set066.csv
test 86 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set059.csv
test 87 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set013.csv
test 88 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set089.csv
test 89 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set071.csv
test 90 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set088.csv
test 91 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set037.csv
test 92 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set022.csv
test 93 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set084.csv
test 94 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set010.csv
test 95 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set097.csv
test 96 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set068.csv
test 97 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set065.csv
test 98 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set048.csv
test 99 /beegfs/ye53nis/saves/firstartifact_Mar2020/traces_brightclust_Mar2020_set045.csv
shapes of feature dataframe: (16384, 8000) and label dataframe: (16384, 8000)
shapes of feature dataframe: (16384, 2000) and label dataframe: (16384, 2000)

for each 20,000 timestap trace there are the following numbers of corrupted timesteps:
 label001_1    16384
label001_1    16384
label001_1    16384
label001_1    16384
label001_1    16384
dtype: int64
2020-05-31 15:49:25.643578: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical opera
tions:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-05-31 15:49:25.653635: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz
2020-05-31 15:49:25.655125: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x556ab2694730 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-31 15:49:25.655169: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
number of training examples: 6400, number of validation examples: 1600

------------------------
number of test examples: 2000

input - shape:   (None, 16384, 1)
output - shape:  (None, 16384, 1)
2020-05-31 15:49:32.395226: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the A
BCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  if not isinstance(wrapped_dict, collections.Mapping):
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Pytho
n 3.0, use inspect.signature() or inspect.getfullargspec()
  all_param_names, _, _, all_default_values = inspect.getargspec(fn)  # pylint: disable=W1505
Epoch 1/2
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from
'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  if not isinstance(values, collections.Sequence):
 1/10 [==>...........................] - ETA: 0s - loss: 0.5648 - tp: 72554.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 9366.0000 - precision: 1.0000 - recall: 0.8857 - accuracy: 0.8857 - auc: 0.0000e+0020
20-05-31 15:49:44.598769: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager
.profiler) is deprecated and will be removed after 2020-07-01.
Instructions for updating:
use `tf.profiler.experimental.stop` instead.
2020-05-31 15:49:46.123954: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_15_49_46
2020-05-31 15:49:46.137599: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.trace.json.gz
2020-05-31 15:49:46.163428: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_15_49_46
2020-05-31 15:49:46.163531: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.memory
_profile.json.gz
2020-05-31 15:49:46.165569: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_15_49_46Dumped tool data for xplane.pb to /tmp/tb/trai
n/plugins/profile/2020_05_31_15_49_46/node171.xplane.pb
Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.overview_page.pb
Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.input_pipeline.pb
Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.tensorflow_stats.pb
Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_15_49_46/node171.kernel_stats.pb

10/10 [==============================] - 23s 2s/step - loss: 0.0638 - tp: 809834.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 9366.0000 - precision: 1.0000 - recall: 0.9886 - accuracy: 0.9886 - auc: 0.0000e
+00 - val_loss: 0.0000e+00 - val_tp: 819200.0000 - val_fp: 0.0000e+00 - val_tn: 0.0000e+00 - val_fn: 0.0000e+00 - val_precision: 1.0000 - val_recall: 1.0000 - val_accuracy: 1.0000 - val_auc: 0.0000e+00
Epoch 2/2
10/10 [==============================] - 18s 2s/step - loss: 6.4955e-17 - tp: 819200.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 0.0000e+00 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 - auc: 0.
0000e+00 - val_loss: 0.0000e+00 - val_tp: 819200.0000 - val_fp: 0.0000e+00 - val_tn: 0.0000e+00 - val_fn: 0.0000e+00 - val_precision: 1.0000 - val_recall: 1.0000 - val_accuracy: 1.0000 - val_auc: 0.0000e+00
400/400 [==============================] - 120s 300ms/step - loss: 0.0000e+00 - tp: 32768000.0000 - fp: 0.0000e+00 - tn: 0.0000e+00 - fn: 0.0000e+00 - precision: 1.0000 - recall: 1.0000 - accuracy: 1.0000 -
 auc: 0.0000e+00
2020/05/31 15:52:36 INFO mlflow.projects: === Run (ID '7f23f6ba7a244914b3cbbebd731d50a1') succeeded ===

These two test runs used the wrong dataset! the firstartifact_Mar2020 dataset is bright clusters, but the label is the uncorrupted trace, not just the artifact information. This dataset is meant for a Variational Autoencoder-Training.

2.1.6 experimental run

mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=70 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact_Sep2019 -P steps_per_epoch=400 -P validation_steps=200
(tensorflow_nightly) [ye53nis@node171 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=70 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact_Sep2019 -
P steps_per_epoch=400 -P validation_steps=200
2020/05/31 21:39:53 INFO mlflow.projects: === Created directory /tmp/tmpnc10f5ut for downloading remote URIs passed to arguments of type 'path' ===
2020/05/31 21:39:53 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/f
luotracify/training/train.py /beegfs/ye53nis/drmed-git/src 5 0.2 16384 None 70 /beegfs/ye53nis/saves/firstartefact_Sep2019 400 200' in run with ID '1aefda1366f04f5da5d1fc2241ad9208' ===
2020-05-31 21:39:54.753597: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No
 such file or directory
2020-05-31 21:39:54.753646: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favo
ur of importlib; see the module's documentation for alternative uses
  import imp
2.3.0-dev20200527
2020-05-31 21:39:57.636018: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file
 or directory
2020-05-31 21:39:57.636052: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-05-31 21:39:57.636075: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node171): /proc/driver/nvidia/version does not exist
GPUs:  []
train 0 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set027.csv
train 1 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set087.csv
train 2 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set003.csv
train 3 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set056.csv
train 4 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set076.csv
train 5 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set094.csv
train 6 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set017.csv
train 7 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set074.csv
train 8 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set055.csv
train 9 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set096.csv
train 10 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set054.csv
train 11 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set093.csv
train 12 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set079.csv
train 13 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set014.csv
train 14 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set008.csv
train 15 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set031.csv
train 16 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set023.csv
train 17 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set025.csv
train 18 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set034.csv
train 19 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set009.csv
train 20 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set044.csv
train 21 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set063.csv
train 22 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set004.csv
train 23 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set072.csv
train 24 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set046.csv
train 25 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set049.csv
train 26 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set007.csv
train 27 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set100.csv
train 28 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set083.csv
train 29 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set077.csv
train 30 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set061.csv
train 31 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set081.csv
train 32 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set091.csv
train 33 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set069.csv
train 34 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set052.csv
train 35 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set028.csv
train 36 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set019.csv
train 37 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set057.csv
train 38 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set064.csv
train 39 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set075.csv
train 40 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set002.csv
train 41 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set062.csv
train 42 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set043.csv
train 43 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set042.csv
train 44 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set005.csv
train 45 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set016.csv
train 46 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set018.csv
train 47 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set041.csv
train 48 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set039.csv
train 49 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set006.csv
train 50 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set092.csv
train 51 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set060.csv
train 52 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set001.csv
train 53 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set035.csv
train 54 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set029.csv
train 55 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set051.csv
train 56 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set012.csv
train 57 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set036.csv
train 58 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set024.csv
train 59 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set053.csv
train 60 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set011.csv
train 61 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set032.csv
train 62 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set067.csv
train 63 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set058.csv
train 64 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set080.csv
train 65 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set086.csv
train 66 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set033.csv
train 67 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set085.csv
train 68 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set015.csv
train 69 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set090.csv
train 70 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set020.csv
train 71 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set030.csv
train 72 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set050.csv
train 73 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set098.csv
train 74 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set099.csv
train 75 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set070.csv
train 76 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set021.csv
train 77 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set095.csv
train 78 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set073.csv
train 79 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set078.csv
test 80 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set026.csv
test 81 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set038.csv
test 82 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set082.csv
test 83 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set047.csv
test 84 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set040.csv
test 85 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set066.csv
test 86 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set059.csv
test 87 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set013.csv
test 88 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set089.csv
test 89 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set071.csv
test 90 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set088.csv
test 91 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set037.csv
test 92 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set022.csv
test 93 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set084.csv
test 94 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set010.csv
test 95 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set097.csv
test 96 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set068.csv
test 97 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set065.csv
test 98 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set048.csv
test 99 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set045.csv
shapes of feature dataframe: (20000, 8000) and label dataframe: (20000, 8000)
shapes of feature dataframe: (20000, 2000) and label dataframe: (20000, 2000)

for each 20,000 timestap trace there are the following numbers of corrupted timesteps:
 label001_1    6286
label001_1    2568
label001_1    4495
label001_1    4414
label001_1    1105
dtype: int64
2020-05-31 21:44:15.811359: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical opera
tions:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-05-31 21:44:15.820877: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz
2020-05-31 21:44:15.822374: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55acbe3932e0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-31 21:44:15.822402: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
number of training examples: 6400, number of validation examples: 1600

------------------------
number of test examples: 2000

input - shape:   (None, 16384, 1)
output - shape:  (None, 16384, 1)
2020-05-31 21:44:19.667769: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the A
BCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  if not isinstance(wrapped_dict, collections.Mapping):
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Pytho
n 3.0, use inspect.signature() or inspect.getfullargspec()
  all_param_names, _, _, all_default_values = inspect.getargspec(fn)  # pylint: disable=W1505
Epoch 1/70
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from
'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  if not isinstance(values, collections.Sequence):
  1/400 [..............................] - ETA: 0s - loss: 1.5974 - tp: 3783.0000 - fp: 23102.0000 - tn: 47580.0000 - fn: 7455.0000 - precision: 0.1407 - recall: 0.3366 - accuracy: 0.6270 - auc: 0.51772020-
05-31 21:44:31.349376: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager
.profiler) is deprecated and will be removed after 2020-07-01.
Instructions for updating:
use `tf.profiler.experimental.stop` instead.
2020-05-31 21:44:32.843703: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_21_44_32
2020-05-31 21:44:32.857345: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_21_44_32/node171.trace.json.gz
2020-05-31 21:44:32.882350: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_21_44_32
2020-05-31 21:44:32.882475: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_05_31_21_44_32/node171.memory
_profile.json.gz
2020-05-31 21:44:32.884516: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_05_31_21_44_32Dumped tool data for xplane.pb to /tmp/tb/trai
n/plugins/profile/2020_05_31_21_44_32/node171.xplane.pb
Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_05_31_21_44_32/node171.overview_page.pb
Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_05_31_21_44_32/node171.input_pipeline.pb
Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_21_44_32/node171.tensorflow_stats.pb
Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_05_31_21_44_32/node171.kernel_stats.pb

400/400 [==============================] - 648s 2s/step - loss: 1.2332 - tp: 3788781.0000 - fp: 1219922.0000 - tn: 24049440.0000 - fn: 3709858.0000 - precision: 0.7564 - recall: 0.5053 - accuracy: 0.8496 -
auc: 0.8489 - val_loss: 322.0566 - val_tp: 3646493.0000 - val_fp: 12737507.0000 - val_tn: 0.0000e+00 - val_fn: 0.0000e+00 - val_precision: 0.2226 - val_recall: 1.0000 - val_accuracy: 0.2226 - val_auc: 0.500
0
Epoch 2/70
400/400 [==============================] - 644s 2s/step - loss: 1.0109 - tp: 5005706.0000 - fp: 1436756.0000 - tn: 23682896.0000 - fn: 2642650.0000 - precision: 0.7770 - recall: 0.6545 - accuracy: 0.8755 -
auc: 0.8979 - val_loss: 606.1023 - val_tp: 3756979.0000 - val_fp: 12626343.0000 - val_tn: 645.0000 - val_fn: 33.0000 - val_precision: 0.2293 - val_recall: 1.0000 - val_accuracy: 0.2293 - val_auc: 0.5000
Epoch 3/70
400/400 [==============================] - 628s 2s/step - loss: 0.8285 - tp: 5051785.0000 - fp: 1374618.0000 - tn: 23855938.0000 - fn: 2485671.0000 - precision: 0.7861 - recall: 0.6702 - accuracy: 0.8822 -
auc: 0.9026 - val_loss: 10.0331 - val_tp: 3878641.0000 - val_fp: 12500019.0000 - val_tn: 4304.0000 - val_fn: 1036.0000 - val_precision: 0.2368 - val_recall: 0.9997 - val_accuracy: 0.2370 - val_auc: 0.5000
Epoch 4/70
400/400 [==============================] - 629s 2s/step - loss: 0.6738 - tp: 5537594.0000 - fp: 1337082.0000 - tn: 23959264.0000 - fn: 1934060.0000 - precision: 0.8055 - recall: 0.7411 - accuracy: 0.9002 -
auc: 0.9303 - val_loss: 1.2653 - val_tp: 3303018.0000 - val_fp: 1870945.0000 - val_tn: 10869556.0000 - val_fn: 340481.0000 - val_precision: 0.6384 - val_recall: 0.9066 - val_accuracy: 0.8650 - val_auc: 0.92
24
Epoch 5/70
400/400 [==============================] - 631s 2s/step - loss: 0.5958 - tp: 5587069.0000 - fp: 1162314.0000 - tn: 24313792.0000 - fn: 1704840.0000 - precision: 0.8278 - recall: 0.7662 - accuracy: 0.9125 -
auc: 0.9409 - val_loss: 1.2564 - val_tp: 2402186.0000 - val_fp: 760086.0000 - val_tn: 11788196.0000 - val_fn: 1433532.0000 - val_precision: 0.7596 - val_recall: 0.6263 - val_accuracy: 0.8661 - val_auc: 0.84
39
Epoch 6/70
400/400 [==============================] - 632s 2s/step - loss: 0.5248 - tp: 6187578.0000 - fp: 1102970.0000 - tn: 24040936.0000 - fn: 1436530.0000 - precision: 0.8487 - recall: 0.8116 - accuracy: 0.9225 -
auc: 0.9536 - val_loss: 3.8886 - val_tp: 3699587.0000 - val_fp: 5786776.0000 - val_tn: 6872632.0000 - val_fn: 25005.0000 - val_precision: 0.3900 - val_recall: 0.9933 - val_accuracy: 0.6453 - val_auc: 0.8590
Epoch 7/70
400/400 [==============================] - 629s 2s/step - loss: 0.5417 - tp: 6075103.0000 - fp: 1198710.0000 - tn: 23976388.0000 - fn: 1517805.0000 - precision: 0.8352 - recall: 0.8001 - accuracy: 0.9171 -
auc: 0.9507 - val_loss: 0.6575 - val_tp: 3549691.0000 - val_fp: 1710464.0000 - val_tn: 10816997.0000 - val_fn: 306848.0000 - val_precision: 0.6748 - val_recall: 0.9204 - val_accuracy: 0.8769 - val_auc: 0.95
25
Epoch 8/70
400/400 [==============================] - 630s 2s/step - loss: 0.5261 - tp: 6041081.0000 - fp: 1000532.0000 - tn: 24306236.0000 - fn: 1420140.0000 - precision: 0.8579 - recall: 0.8097 - accuracy: 0.9261 -
auc: 0.9527 - val_loss: 21.2177 - val_tp: 3715793.0000 - val_fp: 9558636.0000 - val_tn: 3103000.0000 - val_fn: 6571.0000 - val_precision: 0.2799 - val_recall: 0.9982 - val_accuracy: 0.4162 - val_auc: 0.6830
Epoch 9/70
400/400 [==============================] - 634s 2s/step - loss: 0.5777 - tp: 5925253.0000 - fp: 1206548.0000 - tn: 23980040.0000 - fn: 1656156.0000 - precision: 0.8308 - recall: 0.7816 - accuracy: 0.9126 -
auc: 0.9467 - val_loss: 1.4550 - val_tp: 2525888.0000 - val_fp: 1794119.0000 - val_tn: 10974268.0000 - val_fn: 1089725.0000 - val_precision: 0.5847 - val_recall: 0.6986 - val_accuracy: 0.8240 - val_auc: 0.8
157
Epoch 10/70
400/400 [==============================] - 633s 2s/step - loss: 0.4670 - tp: 6137276.0000 - fp: 1085998.0000 - tn: 24173720.0000 - fn: 1370997.0000 - precision: 0.8497 - recall: 0.8174 - accuracy: 0.9250 -
auc: 0.9596 - val_loss: 0.4228 - val_tp: 3267331.0000 - val_fp: 512452.0000 - val_tn: 12134530.0000 - val_fn: 469687.0000 - val_precision: 0.8644 - val_recall: 0.8743 - val_accuracy: 0.9401 - val_auc: 0.958
6
Epoch 11/70
400/400 [==============================] - 636s 2s/step - loss: 0.4183 - tp: 6387249.0000 - fp: 950432.0000 - tn: 24334356.0000 - fn: 1095980.0000 - precision: 0.8705 - recall: 0.8535 - accuracy: 0.9375 - a
uc: 0.9662 - val_loss: 2.5835 - val_tp: 3791404.0000 - val_fp: 10983628.0000 - val_tn: 1606117.0000 - val_fn: 2851.0000 - val_precision: 0.2566 - val_recall: 0.9992 - val_accuracy: 0.3294 - val_auc: 0.9205
Epoch 12/70
400/400 [==============================] - 633s 2s/step - loss: 0.3622 - tp: 6622657.0000 - fp: 875498.0000 - tn: 24381258.0000 - fn: 888583.0000 - precision: 0.8832 - recall: 0.8817 - accuracy: 0.9462 - au
c: 0.9716 - val_loss: 0.3789 - val_tp: 3055309.0000 - val_fp: 146845.0000 - val_tn: 12452879.0000 - val_fn: 728967.0000 - val_precision: 0.9541 - val_recall: 0.8074 - val_accuracy: 0.9465 - val_auc: 0.9659
Epoch 13/70
400/400 [==============================] - 631s 2s/step - loss: 0.3590 - tp: 6660677.0000 - fp: 869994.0000 - tn: 24335574.0000 - fn: 901756.0000 - precision: 0.8845 - recall: 0.8808 - accuracy: 0.9459 - au
c: 0.9722 - val_loss: 0.4284 - val_tp: 2585078.0000 - val_fp: 26120.0000 - val_tn: 12713922.0000 - val_fn: 1058880.0000 - val_precision: 0.9900 - val_recall: 0.7094 - val_accuracy: 0.9338 - val_auc: 0.9595
Epoch 14/70
400/400 [==============================] - 631s 2s/step - loss: 0.3391 - tp: 6632643.0000 - fp: 838058.0000 - tn: 24461996.0000 - fn: 835290.0000 - precision: 0.8878 - recall: 0.8881 - accuracy: 0.9489 - au
c: 0.9747 - val_loss: 0.3533 - val_tp: 3003363.0000 - val_fp: 127052.0000 - val_tn: 12492719.0000 - val_fn: 760866.0000 - val_precision: 0.9594 - val_recall: 0.7979 - val_accuracy: 0.9458 - val_auc: 0.9766
Epoch 15/70
400/400 [==============================] - 634s 2s/step - loss: 0.3422 - tp: 6682203.0000 - fp: 861078.0000 - tn: 24351496.0000 - fn: 873232.0000 - precision: 0.8858 - recall: 0.8844 - accuracy: 0.9471 - au
c: 0.9741 - val_loss: 0.3630 - val_tp: 2919284.0000 - val_fp: 89104.0000 - val_tn: 12597566.0000 - val_fn: 778046.0000 - val_precision: 0.9704 - val_recall: 0.7896 - val_accuracy: 0.9471 - val_auc: 0.9763
Epoch 16/70
400/400 [==============================] - 632s 2s/step - loss: 0.3356 - tp: 6670288.0000 - fp: 818050.0000 - tn: 24453228.0000 - fn: 826432.0000 - precision: 0.8908 - recall: 0.8898 - accuracy: 0.9498 - au
c: 0.9748 - val_loss: 0.7556 - val_tp: 1874704.0000 - val_fp: 1072.0000 - val_tn: 12589343.0000 - val_fn: 1918881.0000 - val_precision: 0.9994 - val_recall: 0.4942 - val_accuracy: 0.8828 - val_auc: 0.9213
Epoch 17/70
400/400 [==============================] - 629s 2s/step - loss: 0.3260 - tp: 6760875.0000 - fp: 795293.0000 - tn: 24413246.0000 - fn: 798585.0000 - precision: 0.8947 - recall: 0.8944 - accuracy: 0.9514 - au
c: 0.9761 - val_loss: 0.4649 - val_tp: 2671759.0000 - val_fp: 36166.0000 - val_tn: 12592753.0000 - val_fn: 1083322.0000 - val_precision: 0.9866 - val_recall: 0.7115 - val_accuracy: 0.9317 - val_auc: 0.9546
Epoch 18/70
400/400 [==============================] - 631s 2s/step - loss: 0.3309 - tp: 6559233.0000 - fp: 790466.0000 - tn: 24604904.0000 - fn: 813389.0000 - precision: 0.8924 - recall: 0.8897 - accuracy: 0.9511 - au
c: 0.9751 - val_loss: 0.4767 - val_tp: 2812359.0000 - val_fp: 90282.0000 - val_tn: 12389838.0000 - val_fn: 1091521.0000 - val_precision: 0.9689 - val_recall: 0.7204 - val_accuracy: 0.9279 - val_auc: 0.9586
Epoch 19/70
400/400 [==============================] - 631s 2s/step - loss: 0.3375 - tp: 6783791.0000 - fp: 866853.0000 - tn: 24263056.0000 - fn: 854305.0000 - precision: 0.8867 - recall: 0.8882 - accuracy: 0.9475 - au
c: 0.9752 - val_loss: 0.3983 - val_tp: 2551077.0000 - val_fp: 35530.0000 - val_tn: 12721430.0000 - val_fn: 1075963.0000 - val_precision: 0.9863 - val_recall: 0.7033 - val_accuracy: 0.9322 - val_auc: 0.9710
Epoch 20/70
400/400 [==============================] - 632s 2s/step - loss: 0.3254 - tp: 6564303.0000 - fp: 830891.0000 - tn: 24563264.0000 - fn: 809541.0000 - precision: 0.8876 - recall: 0.8902 - accuracy: 0.9499 - au
c: 0.9757 - val_loss: 0.5314 - val_tp: 2046949.0000 - val_fp: 3561.0000 - val_tn: 12714086.0000 - val_fn: 1619404.0000 - val_precision: 0.9983 - val_recall: 0.5583 - val_accuracy: 0.9009 - val_auc: 0.9545
Epoch 21/70
400/400 [==============================] - 640s 2s/step - loss: 0.3196 - tp: 6686380.0000 - fp: 775563.0000 - tn: 24496560.0000 - fn: 809502.0000 - precision: 0.8961 - recall: 0.8920 - accuracy: 0.9516 - au
c: 0.9765 - val_loss: 0.4055 - val_tp: 2871943.0000 - val_fp: 218969.0000 - val_tn: 12467101.0000 - val_fn: 825987.0000 - val_precision: 0.9292 - val_recall: 0.7766 - val_accuracy: 0.9362 - val_auc: 0.9714
Epoch 22/70
400/400 [==============================] - 625s 2s/step - loss: 0.3154 - tp: 6756372.0000 - fp: 777461.0000 - tn: 24450556.0000 - fn: 783609.0000 - precision: 0.8968 - recall: 0.8961 - accuracy: 0.9524 - au
c: 0.9770 - val_loss: 0.3362 - val_tp: 3143955.0000 - val_fp: 135752.0000 - val_tn: 12411791.0000 - val_fn: 692502.0000 - val_precision: 0.9586 - val_recall: 0.8195 - val_accuracy: 0.9494 - val_auc: 0.9754
Epoch 23/70
400/400 [==============================] - 633s 2s/step - loss: 0.3294 - tp: 6975026.0000 - fp: 773797.0000 - tn: 24208046.0000 - fn: 811140.0000 - precision: 0.9001 - recall: 0.8958 - accuracy: 0.9516 - au
c: 0.9760 - val_loss: 0.3650 - val_tp: 2827380.0000 - val_fp: 88109.0000 - val_tn: 12626725.0000 - val_fn: 841786.0000 - val_precision: 0.9698 - val_recall: 0.7706 - val_accuracy: 0.9432 - val_auc: 0.9697
Epoch 24/70
400/400 [==============================] - 629s 2s/step - loss: 0.3184 - tp: 6701853.0000 - fp: 767247.0000 - tn: 24524388.0000 - fn: 774508.0000 - precision: 0.8973 - recall: 0.8964 - accuracy: 0.9529 - au
c: 0.9767 - val_loss: 0.5394 - val_tp: 2252899.0000 - val_fp: 4050.0000 - val_tn: 12590875.0000 - val_fn: 1536176.0000 - val_precision: 0.9982 - val_recall: 0.5946 - val_accuracy: 0.9060 - val_auc: 0.9528
Epoch 25/70
400/400 [==============================] - 633s 2s/step - loss: 0.3189 - tp: 6642243.0000 - fp: 777194.0000 - tn: 24527812.0000 - fn: 820758.0000 - precision: 0.8952 - recall: 0.8900 - accuracy: 0.9512 - au
c: 0.9762 - val_loss: 0.3377 - val_tp: 3281478.0000 - val_fp: 329375.0000 - val_tn: 12262368.0000 - val_fn: 510779.0000 - val_precision: 0.9088 - val_recall: 0.8653 - val_accuracy: 0.9487 - val_auc: 0.9760
Epoch 26/70
400/400 [==============================] - 636s 2s/step - loss: 0.3137 - tp: 6689776.0000 - fp: 788613.0000 - tn: 24505860.0000 - fn: 783748.0000 - precision: 0.8945 - recall: 0.8951 - accuracy: 0.9520 - au
c: 0.9772 - val_loss: 0.4429 - val_tp: 2772106.0000 - val_fp: 45921.0000 - val_tn: 12444722.0000 - val_fn: 1121251.0000 - val_precision: 0.9837 - val_recall: 0.7120 - val_accuracy: 0.9288 - val_auc: 0.9625
Epoch 27/70
400/400 [==============================] - 632s 2s/step - loss: 0.3110 - tp: 6754135.0000 - fp: 761746.0000 - tn: 24463904.0000 - fn: 788220.0000 - precision: 0.8986 - recall: 0.8955 - accuracy: 0.9527 - au
c: 0.9778 - val_loss: 0.3217 - val_tp: 3322329.0000 - val_fp: 341816.0000 - val_tn: 12309980.0000 - val_fn: 409875.0000 - val_precision: 0.9067 - val_recall: 0.8902 - val_accuracy: 0.9541 - val_auc: 0.9829
Epoch 28/70
400/400 [==============================] - 632s 2s/step - loss: 0.3143 - tp: 6711103.0000 - fp: 764757.0000 - tn: 24512308.0000 - fn: 779835.0000 - precision: 0.8977 - recall: 0.8959 - accuracy: 0.9529 - au
c: 0.9772 - val_loss: 0.5518 - val_tp: 2137267.0000 - val_fp: 8417.0000 - val_tn: 12762333.0000 - val_fn: 1475983.0000 - val_precision: 0.9961 - val_recall: 0.5915 - val_accuracy: 0.9094 - val_auc: 0.9459
Epoch 29/70
400/400 [==============================] - 636s 2s/step - loss: 0.3172 - tp: 6808597.0000 - fp: 762146.0000 - tn: 24410300.0000 - fn: 786963.0000 - precision: 0.8993 - recall: 0.8964 - accuracy: 0.9527 - au
c: 0.9773 - val_loss: 0.3880 - val_tp: 3284429.0000 - val_fp: 149353.0000 - val_tn: 12279814.0000 - val_fn: 670404.0000 - val_precision: 0.9565 - val_recall: 0.8305 - val_accuracy: 0.9500 - val_auc: 0.9824
Epoch 30/70
400/400 [==============================] - 636s 2s/step - loss: 0.3119 - tp: 6732407.0000 - fp: 734825.0000 - tn: 24527008.0000 - fn: 773770.0000 - precision: 0.9016 - recall: 0.8969 - accuracy: 0.9540 - au
c: 0.9770 - val_loss: 0.3304 - val_tp: 3197972.0000 - val_fp: 378897.0000 - val_tn: 12377137.0000 - val_fn: 429994.0000 - val_precision: 0.8941 - val_recall: 0.8815 - val_accuracy: 0.9506 - val_auc: 0.9788
Epoch 31/70
400/400 [==============================] - 639s 2s/step - loss: 0.3172 - tp: 6584008.0000 - fp: 749012.0000 - tn: 24635250.0000 - fn: 799720.0000 - precision: 0.8979 - recall: 0.8917 - accuracy: 0.9527 - au
c: 0.9763 - val_loss: 0.4527 - val_tp: 3349009.0000 - val_fp: 338939.0000 - val_tn: 12288817.0000 - val_fn: 407235.0000 - val_precision: 0.9081 - val_recall: 0.8916 - val_accuracy: 0.9545 - val_auc: 0.9822
Epoch 32/70
400/400 [==============================] - 632s 2s/step - loss: 0.3146 - tp: 6690763.0000 - fp: 766270.0000 - tn: 24531178.0000 - fn: 779790.0000 - precision: 0.8972 - recall: 0.8956 - accuracy: 0.9528 - au
c: 0.9763 - val_loss: 0.3255 - val_tp: 2945281.0000 - val_fp: 103765.0000 - val_tn: 12623298.0000 - val_fn: 711656.0000 - val_precision: 0.9660 - val_recall: 0.8054 - val_accuracy: 0.9502 - val_auc: 0.9761
Epoch 33/70
400/400 [==============================] - 632s 2s/step - loss: 0.2975 - tp: 6744548.0000 - fp: 747413.0000 - tn: 24547794.0000 - fn: 728248.0000 - precision: 0.9002 - recall: 0.9025 - accuracy: 0.9550 - au
c: 0.9787 - val_loss: 0.3461 - val_tp: 2865337.0000 - val_fp: 80568.0000 - val_tn: 12713439.0000 - val_fn: 724656.0000 - val_precision: 0.9727 - val_recall: 0.7981 - val_accuracy: 0.9509 - val_auc: 0.9780
Epoch 34/70
400/400 [==============================] - 635s 2s/step - loss: 0.3060 - tp: 6610739.0000 - fp: 717355.0000 - tn: 24665928.0000 - fn: 773982.0000 - precision: 0.9021 - recall: 0.8952 - accuracy: 0.9545 - au
c: 0.9768 - val_loss: 0.3895 - val_tp: 2875845.0000 - val_fp: 64187.0000 - val_tn: 12593693.0000 - val_fn: 850275.0000 - val_precision: 0.9782 - val_recall: 0.7718 - val_accuracy: 0.9442 - val_auc: 0.9762
Epoch 35/70
400/400 [==============================] - 632s 2s/step - loss: 0.3131 - tp: 6658644.0000 - fp: 787948.0000 - tn: 24555564.0000 - fn: 765835.0000 - precision: 0.8942 - recall: 0.8968 - accuracy: 0.9526 - au
c: 0.9770 - val_loss: 0.3298 - val_tp: 3291381.0000 - val_fp: 128257.0000 - val_tn: 12266428.0000 - val_fn: 697934.0000 - val_precision: 0.9625 - val_recall: 0.8250 - val_accuracy: 0.9496 - val_auc: 0.9787
Epoch 36/70
400/400 [==============================] - 631s 2s/step - loss: 0.3076 - tp: 6714273.0000 - fp: 757539.0000 - tn: 24536108.0000 - fn: 760070.0000 - precision: 0.8986 - recall: 0.8983 - accuracy: 0.9537 - au
c: 0.9776 - val_loss: 0.3586 - val_tp: 2828629.0000 - val_fp: 70731.0000 - val_tn: 12691571.0000 - val_fn: 793069.0000 - val_precision: 0.9756 - val_recall: 0.7810 - val_accuracy: 0.9473 - val_auc: 0.9764
Epoch 37/70
400/400 [==============================] - 630s 2s/step - loss: 0.3019 - tp: 6673190.0000 - fp: 759238.0000 - tn: 24567992.0000 - fn: 767563.0000 - precision: 0.8978 - recall: 0.8968 - accuracy: 0.9534 - au
c: 0.9783 - val_loss: 0.3614 - val_tp: 3049928.0000 - val_fp: 93276.0000 - val_tn: 12445161.0000 - val_fn: 795635.0000 - val_precision: 0.9703 - val_recall: 0.7931 - val_accuracy: 0.9457 - val_auc: 0.9755
Epoch 38/70
400/400 [==============================] - 631s 2s/step - loss: 0.3022 - tp: 6902334.0000 - fp: 772344.0000 - tn: 24353052.0000 - fn: 740273.0000 - precision: 0.8994 - recall: 0.9031 - accuracy: 0.9538 - au
c: 0.9785 - val_loss: 0.3220 - val_tp: 2932889.0000 - val_fp: 104394.0000 - val_tn: 12669356.0000 - val_fn: 677361.0000 - val_precision: 0.9656 - val_recall: 0.8124 - val_accuracy: 0.9523 - val_auc: 0.9783
Epoch 39/70
400/400 [==============================] - 632s 2s/step - loss: 0.2978 - tp: 6800883.0000 - fp: 734721.0000 - tn: 24491858.0000 - fn: 740536.0000 - precision: 0.9025 - recall: 0.9018 - accuracy: 0.9550 - au
c: 0.9785 - val_loss: 0.3295 - val_tp: 3082233.0000 - val_fp: 128611.0000 - val_tn: 12542874.0000 - val_fn: 630282.0000 - val_precision: 0.9599 - val_recall: 0.8302 - val_accuracy: 0.9537 - val_auc: 0.9799
Epoch 40/70
400/400 [==============================] - 634s 2s/step - loss: 0.3083 - tp: 6769252.0000 - fp: 762049.0000 - tn: 24472648.0000 - fn: 764043.0000 - precision: 0.8988 - recall: 0.8986 - accuracy: 0.9534 - au
c: 0.9771 - val_loss: 0.3593 - val_tp: 2826188.0000 - val_fp: 59581.0000 - val_tn: 12671875.0000 - val_fn: 826356.0000 - val_precision: 0.9794 - val_recall: 0.7738 - val_accuracy: 0.9459 - val_auc: 0.9756
Epoch 41/70
400/400 [==============================] - 630s 2s/step - loss: 0.2953 - tp: 6718090.0000 - fp: 747344.0000 - tn: 24561188.0000 - fn: 741388.0000 - precision: 0.8999 - recall: 0.9006 - accuracy: 0.9546 - au
c: 0.9793 - val_loss: 0.3628 - val_tp: 2996938.0000 - val_fp: 97647.0000 - val_tn: 12518986.0000 - val_fn: 770429.0000 - val_precision: 0.9684 - val_recall: 0.7955 - val_accuracy: 0.9470 - val_auc: 0.9774
Epoch 42/70
400/400 [==============================] - 629s 2s/step - loss: 0.3039 - tp: 6654758.0000 - fp: 731543.0000 - tn: 24611838.0000 - fn: 769861.0000 - precision: 0.9010 - recall: 0.8963 - accuracy: 0.9542 - au
c: 0.9775 - val_loss: 0.3282 - val_tp: 2923757.0000 - val_fp: 81924.0000 - val_tn: 12615035.0000 - val_fn: 763284.0000 - val_precision: 0.9727 - val_recall: 0.7930 - val_accuracy: 0.9484 - val_auc: 0.9759
Epoch 43/70
400/400 [==============================] - 636s 2s/step - loss: 0.2937 - tp: 6435979.0000 - fp: 733398.0000 - tn: 24870358.0000 - fn: 728270.0000 - precision: 0.8977 - recall: 0.8983 - accuracy: 0.9554 - au
c: 0.9781 - val_loss: 0.3116 - val_tp: 3021625.0000 - val_fp: 126888.0000 - val_tn: 12599400.0000 - val_fn: 636087.0000 - val_precision: 0.9597 - val_recall: 0.8261 - val_accuracy: 0.9534 - val_auc: 0.9791
Epoch 44/70
400/400 [==============================] - 638s 2s/step - loss: 0.3207 - tp: 6810749.0000 - fp: 779963.0000 - tn: 24383824.0000 - fn: 793469.0000 - precision: 0.8972 - recall: 0.8957 - accuracy: 0.9520 - au
c: 0.9763 - val_loss: 0.3609 - val_tp: 3046190.0000 - val_fp: 96429.0000 - val_tn: 12529182.0000 - val_fn: 712199.0000 - val_precision: 0.9693 - val_recall: 0.8105 - val_accuracy: 0.9506 - val_auc: 0.9784
Epoch 45/70
400/400 [==============================] - 631s 2s/step - loss: 0.3025 - tp: 7086768.0000 - fp: 750575.0000 - tn: 24182492.0000 - fn: 748166.0000 - precision: 0.9042 - recall: 0.9045 - accuracy: 0.9543 - au
c: 0.9790 - val_loss: 0.3397 - val_tp: 2823520.0000 - val_fp: 64000.0000 - val_tn: 12722924.0000 - val_fn: 773556.0000 - val_precision: 0.9778 - val_recall: 0.7849 - val_accuracy: 0.9489 - val_auc: 0.9768
Epoch 46/70
400/400 [==============================] - 639s 2s/step - loss: 0.3029 - tp: 6575419.0000 - fp: 752434.0000 - tn: 24679216.0000 - fn: 760935.0000 - precision: 0.8973 - recall: 0.8963 - accuracy: 0.9538 - au
c: 0.9777 - val_loss: 0.3449 - val_tp: 3214822.0000 - val_fp: 77887.0000 - val_tn: 12327207.0000 - val_fn: 764084.0000 - val_precision: 0.9763 - val_recall: 0.8080 - val_accuracy: 0.9486 - val_auc: 0.9800
Epoch 47/70
400/400 [==============================] - 638s 2s/step - loss: 0.2988 - tp: 7041341.0000 - fp: 779154.0000 - tn: 24212120.0000 - fn: 735391.0000 - precision: 0.9004 - recall: 0.9054 - accuracy: 0.9538 - au
c: 0.9794 - val_loss: 0.3507 - val_tp: 2992658.0000 - val_fp: 83764.0000 - val_tn: 12550469.0000 - val_fn: 757109.0000 - val_precision: 0.9728 - val_recall: 0.7981 - val_accuracy: 0.9487 - val_auc: 0.9795
Epoch 48/70
400/400 [==============================] - 633s 2s/step - loss: 0.3015 - tp: 6740728.0000 - fp: 737863.0000 - tn: 24521192.0000 - fn: 768220.0000 - precision: 0.9013 - recall: 0.8977 - accuracy: 0.9540 - au
c: 0.9781 - val_loss: 0.3456 - val_tp: 3135590.0000 - val_fp: 126895.0000 - val_tn: 12407028.0000 - val_fn: 714487.0000 - val_precision: 0.9611 - val_recall: 0.8144 - val_accuracy: 0.9486 - val_auc: 0.9755
Epoch 49/70
400/400 [==============================] - 630s 2s/step - loss: 0.2965 - tp: 6929622.0000 - fp: 773455.0000 - tn: 24334720.0000 - fn: 730210.0000 - precision: 0.8996 - recall: 0.9047 - accuracy: 0.9541 - au
c: 0.9793 - val_loss: 0.3253 - val_tp: 2923894.0000 - val_fp: 87045.0000 - val_tn: 12668721.0000 - val_fn: 704340.0000 - val_precision: 0.9711 - val_recall: 0.8059 - val_accuracy: 0.9517 - val_auc: 0.9786
Epoch 50/70
400/400 [==============================] - 631s 2s/step - loss: 0.3053 - tp: 6698368.0000 - fp: 772097.0000 - tn: 24544316.0000 - fn: 753212.0000 - precision: 0.8966 - recall: 0.8989 - accuracy: 0.9535 - au
c: 0.9778 - val_loss: 0.3687 - val_tp: 2958570.0000 - val_fp: 52523.0000 - val_tn: 12513477.0000 - val_fn: 859430.0000 - val_precision: 0.9826 - val_recall: 0.7749 - val_accuracy: 0.9443 - val_auc: 0.9771
Epoch 51/70
400/400 [==============================] - 635s 2s/step - loss: 0.2971 - tp: 6664028.0000 - fp: 737094.0000 - tn: 24626012.0000 - fn: 740867.0000 - precision: 0.9004 - recall: 0.8999 - accuracy: 0.9549 - au
c: 0.9786 - val_loss: 0.3459 - val_tp: 3029844.0000 - val_fp: 116426.0000 - val_tn: 12539360.0000 - val_fn: 698370.0000 - val_precision: 0.9630 - val_recall: 0.8127 - val_accuracy: 0.9503 - val_auc: 0.9793
Epoch 52/70
400/400 [==============================] - 633s 2s/step - loss: 0.3026 - tp: 6753347.0000 - fp: 678789.0000 - tn: 24527348.0000 - fn: 808513.0000 - precision: 0.9087 - recall: 0.8931 - accuracy: 0.9546 - au
c: 0.9785 - val_loss: 0.3330 - val_tp: 2851126.0000 - val_fp: 109294.0000 - val_tn: 12699895.0000 - val_fn: 723685.0000 - val_precision: 0.9631 - val_recall: 0.7976 - val_accuracy: 0.9492 - val_auc: 0.9762
Epoch 53/70
400/400 [==============================] - 636s 2s/step - loss: 0.2980 - tp: 6803599.0000 - fp: 685524.0000 - tn: 24488690.0000 - fn: 790195.0000 - precision: 0.9085 - recall: 0.8959 - accuracy: 0.9550 - au
c: 0.9786 - val_loss: 0.3384 - val_tp: 2978082.0000 - val_fp: 84198.0000 - val_tn: 12570606.0000 - val_fn: 751114.0000 - val_precision: 0.9725 - val_recall: 0.7986 - val_accuracy: 0.9490 - val_auc: 0.9782
Epoch 54/70
400/400 [==============================] - 636s 2s/step - loss: 0.3008 - tp: 6617352.0000 - fp: 660115.0000 - tn: 24672528.0000 - fn: 818008.0000 - precision: 0.9093 - recall: 0.8900 - accuracy: 0.9549 - au
c: 0.9776 - val_loss: 0.3487 - val_tp: 3069049.0000 - val_fp: 100663.0000 - val_tn: 12450296.0000 - val_fn: 763992.0000 - val_precision: 0.9682 - val_recall: 0.8007 - val_accuracy: 0.9472 - val_auc: 0.9767
Epoch 55/70
400/400 [==============================] - 634s 2s/step - loss: 0.3041 - tp: 6727196.0000 - fp: 696140.0000 - tn: 24540392.0000 - fn: 804270.0000 - precision: 0.9062 - recall: 0.8932 - accuracy: 0.9542 - au
c: 0.9781 - val_loss: 0.3201 - val_tp: 2941382.0000 - val_fp: 99527.0000 - val_tn: 12647545.0000 - val_fn: 695546.0000 - val_precision: 0.9673 - val_recall: 0.8088 - val_accuracy: 0.9515 - val_auc: 0.9784
Epoch 56/70
400/400 [==============================] - 636s 2s/step - loss: 0.2960 - tp: 6826521.0000 - fp: 708283.0000 - tn: 24472022.0000 - fn: 761183.0000 - precision: 0.9060 - recall: 0.8997 - accuracy: 0.9552 - au
c: 0.9793 - val_loss: 0.3209 - val_tp: 2942017.0000 - val_fp: 95854.0000 - val_tn: 12658328.0000 - val_fn: 687801.0000 - val_precision: 0.9684 - val_recall: 0.8105 - val_accuracy: 0.9522 - val_auc: 0.9787
Epoch 57/70
400/400 [==============================] - 638s 2s/step - loss: 0.2978 - tp: 6648554.0000 - fp: 686959.0000 - tn: 24654000.0000 - fn: 778489.0000 - precision: 0.9064 - recall: 0.8952 - accuracy: 0.9553 - au
c: 0.9782 - val_loss: 0.3418 - val_tp: 3047748.0000 - val_fp: 90564.0000 - val_tn: 12482669.0000 - val_fn: 763019.0000 - val_precision: 0.9711 - val_recall: 0.7998 - val_accuracy: 0.9479 - val_auc: 0.9784
Epoch 58/70
400/400 [==============================] - 637s 2s/step - loss: 0.3043 - tp: 6628231.0000 - fp: 693416.0000 - tn: 24639726.0000 - fn: 806627.0000 - precision: 0.9053 - recall: 0.8915 - accuracy: 0.9542 - au
c: 0.9777 - val_loss: 0.3313 - val_tp: 3159765.0000 - val_fp: 106822.0000 - val_tn: 12398325.0000 - val_fn: 719088.0000 - val_precision: 0.9673 - val_recall: 0.8146 - val_accuracy: 0.9496 - val_auc: 0.9780
Epoch 59/70
400/400 [==============================] - 637s 2s/step - loss: 0.3006 - tp: 6652559.0000 - fp: 718848.0000 - tn: 24625476.0000 - fn: 771096.0000 - precision: 0.9025 - recall: 0.8961 - accuracy: 0.9545 - au
c: 0.9786 - val_loss: 0.3533 - val_tp: 3040082.0000 - val_fp: 82199.0000 - val_tn: 12530147.0000 - val_fn: 731572.0000 - val_precision: 0.9737 - val_recall: 0.8060 - val_accuracy: 0.9503 - val_auc: 0.9795
Epoch 60/70
400/400 [==============================] - 634s 2s/step - loss: 0.3137 - tp: 6891311.0000 - fp: 715443.0000 - tn: 24333106.0000 - fn: 828145.0000 - precision: 0.9059 - recall: 0.8927 - accuracy: 0.9529 - au
c: 0.9774 - val_loss: 0.3250 - val_tp: 2930627.0000 - val_fp: 99442.0000 - val_tn: 12670646.0000 - val_fn: 683285.0000 - val_precision: 0.9672 - val_recall: 0.8109 - val_accuracy: 0.9522 - val_auc: 0.9785
Epoch 61/70
400/400 [==============================] - 641s 2s/step - loss: 0.3028 - tp: 6738706.0000 - fp: 737970.0000 - tn: 24529770.0000 - fn: 761545.0000 - precision: 0.9013 - recall: 0.8985 - accuracy: 0.9542 - au
c: 0.9783 - val_loss: 0.3347 - val_tp: 3211606.0000 - val_fp: 100010.0000 - val_tn: 12350882.0000 - val_fn: 721502.0000 - val_precision: 0.9698 - val_recall: 0.8166 - val_accuracy: 0.9499 - val_auc: 0.9791
Epoch 62/70
400/400 [==============================] - 635s 2s/step - loss: 0.3047 - tp: 6492872.0000 - fp: 710243.0000 - tn: 24764572.0000 - fn: 800309.0000 - precision: 0.9014 - recall: 0.8903 - accuracy: 0.9539 - au
c: 0.9771 - val_loss: 0.3868 - val_tp: 3110246.0000 - val_fp: 71777.0000 - val_tn: 12369449.0000 - val_fn: 832528.0000 - val_precision: 0.9774 - val_recall: 0.7888 - val_accuracy: 0.9448 - val_auc: 0.9764
Epoch 63/70
400/400 [==============================] - 633s 2s/step - loss: 0.3033 - tp: 6979600.0000 - fp: 720858.0000 - tn: 24300348.0000 - fn: 767184.0000 - precision: 0.9064 - recall: 0.9010 - accuracy: 0.9546 - au
c: 0.9788 - val_loss: 0.3340 - val_tp: 3093435.0000 - val_fp: 85835.0000 - val_tn: 12442850.0000 - val_fn: 761880.0000 - val_precision: 0.9730 - val_recall: 0.8024 - val_accuracy: 0.9483 - val_auc: 0.9784
Epoch 64/70
400/400 [==============================] - 637s 2s/step - loss: 0.2992 - tp: 6684955.0000 - fp: 717935.0000 - tn: 24597880.0000 - fn: 767241.0000 - precision: 0.9030 - recall: 0.8970 - accuracy: 0.9547 - au
c: 0.9787 - val_loss: 0.3498 - val_tp: 2864609.0000 - val_fp: 82878.0000 - val_tn: 12711063.0000 - val_fn: 725450.0000 - val_precision: 0.9719 - val_recall: 0.7979 - val_accuracy: 0.9507 - val_auc: 0.9786
Epoch 65/70
400/400 [==============================] - 637s 2s/step - loss: 0.2992 - tp: 6881058.0000 - fp: 713083.0000 - tn: 24387492.0000 - fn: 786363.0000 - precision: 0.9061 - recall: 0.8974 - accuracy: 0.9542 - au
c: 0.9792 - val_loss: 0.3161 - val_tp: 3129890.0000 - val_fp: 123769.0000 - val_tn: 12482708.0000 - val_fn: 647633.0000 - val_precision: 0.9620 - val_recall: 0.8286 - val_accuracy: 0.9529 - val_auc: 0.9799
Epoch 66/70
400/400 [==============================] - 638s 2s/step - loss: 0.2998 - tp: 6513297.0000 - fp: 696712.0000 - tn: 24803000.0000 - fn: 754991.0000 - precision: 0.9034 - recall: 0.8961 - accuracy: 0.9557 - au
c: 0.9779 - val_loss: 0.3222 - val_tp: 3039732.0000 - val_fp: 95061.0000 - val_tn: 12541210.0000 - val_fn: 707997.0000 - val_precision: 0.9697 - val_recall: 0.8111 - val_accuracy: 0.9510 - val_auc: 0.9783
Epoch 67/70
400/400 [==============================] - 635s 2s/step - loss: 0.3011 - tp: 6869971.0000 - fp: 709956.0000 - tn: 24396884.0000 - fn: 791177.0000 - precision: 0.9063 - recall: 0.8967 - accuracy: 0.9542 - au
c: 0.9787 - val_loss: 0.3435 - val_tp: 3137473.0000 - val_fp: 93385.0000 - val_tn: 12425599.0000 - val_fn: 727543.0000 - val_precision: 0.9711 - val_recall: 0.8118 - val_accuracy: 0.9499 - val_auc: 0.9791
Epoch 68/70
400/400 [==============================] - 632s 2s/step - loss: 0.3034 - tp: 6640807.0000 - fp: 757137.0000 - tn: 24604356.0000 - fn: 765698.0000 - precision: 0.8977 - recall: 0.8966 - accuracy: 0.9535 - au
c: 0.9781 - val_loss: 0.3824 - val_tp: 3081203.0000 - val_fp: 98009.0000 - val_tn: 12448722.0000 - val_fn: 756066.0000 - val_precision: 0.9692 - val_recall: 0.8030 - val_accuracy: 0.9479 - val_auc: 0.9776
Epoch 69/70
400/400 [==============================] - 636s 2s/step - loss: 0.2988 - tp: 6698533.0000 - fp: 734236.0000 - tn: 24588786.0000 - fn: 746437.0000 - precision: 0.9012 - recall: 0.8997 - accuracy: 0.9548 - au
c: 0.9788 - val_loss: 0.3604 - val_tp: 2967778.0000 - val_fp: 93005.0000 - val_tn: 12540155.0000 - val_fn: 783062.0000 - val_precision: 0.9696 - val_recall: 0.7912 - val_accuracy: 0.9465 - val_auc: 0.9780
Epoch 70/70
400/400 [==============================] - 638s 2s/step - loss: 0.2996 - tp: 6725869.0000 - fp: 711970.0000 - tn: 24565164.0000 - fn: 765019.0000 - precision: 0.9043 - recall: 0.8979 - accuracy: 0.9549 - au
c: 0.9786 - val_loss: 0.3505 - val_tp: 3050870.0000 - val_fp: 101786.0000 - val_tn: 12476995.0000 - val_fn: 754349.0000 - val_precision: 0.9677 - val_recall: 0.8018 - val_accuracy: 0.9477 - val_auc: 0.9771
400/400 [==============================] - 121s 301ms/step - loss: 0.3684 - tp: 6266358.0000 - fp: 308665.0000 - tn: 24781578.0000 - fn: 1411378.0000 - precision: 0.9531 - recall: 0.8162 - accuracy: 0.9475
- auc: 0.9778
2020/06/01 10:07:50 INFO mlflow.projects: === Run (ID '1aefda1366f04f5da5d1fc2241ad9208') succeeded ===
(tensorflow_nightly) [ye53nis@node171 drmed-git]$
mlflow ui
(tensorflow_nightly) [ye53nis@node171 drmed-git]$ mlflow ui
[2020-06-01 10:38:40 +0200] [298787] [INFO] Starting gunicorn 20.0.4
[2020-06-01 10:38:40 +0200] [298787] [INFO] Listening at: http://127.0.0.1:5000 (298787)
[2020-06-01 10:38:40 +0200] [298787] [INFO] Using worker: sync
[2020-06-01 10:38:40 +0200] [298793] [INFO] Booting worker with pid: 298793

2.1.7 read out logs

2.1.7.1 read out mlflow logs using CLI
conda activate tensorflow_env
cd Programme/drmed-git
export MLFLOW_EXPERIMENT_NAME=exp-devtest
export MLFLOW_TRACKING_URI=file:./data/mlruns
mlflow experiments list
mlflow runs list --experiment-id 1
mlflow artifacts list -r 47b870b8fdcb4445956635c6758caff3
mlflow artifacts list -r 47b870b8fdcb4445956635c6758caff3 -a model
mlflow artifacts list -r 47b870b8fdcb4445956635c6758caff3 -a model_summary.txt
mlflow artifacts list -r 47b870b8fdcb4445956635c6758caff3 -a tensorboard_logs/train/
mlflow artifacts download -r 47b870b8fdcb4445956635c6758caff3
mlflow artifacts download -r 47b870b8fdcb4445956635c6758caff3 -a model
mlflow artifacts download -r 47b870b8fdcb4445956635c6758caff3 -a model_summary.txt
mlflow artifacts download -r 47b870b8fdcb4445956635c6758caff3 -a tensorboard_logs
mlflow runs describe --run-id 47b870b8fdcb4445956635c6758caff3
tensorboard --logdir=data/mlruns/1/47b870b8fdcb4445956635c6758caff3/artifacts/tensorboard_logs
mlflow ui --backend-store-uri file:///home/lex/Programme/drmed-git/data/mlruns
2.1.7.2 Reading out mlflow logs with Python API

I started local machine:

import mlflow
import pprint
%cd /home/lex/Programme/drmed-git/
/home/lex/Programme/drmed-git
uri = 'file:///home/lex/Programme/drmed-git/data/mlruns'
mlflow.set_tracking_uri(uri)
runs = mlflow.search_runs(experiment_ids='0')
print('run_id of first in list: ', runs.iloc[0].run_id)
print('no of runs in list: ', len(runs))
print()
runs
run_id of first in list:  1aefda1366f04f5da5d1fc2241ad9208
no of runs in list:  1
  run\id experiment\id status artifact\uri start\time end\time metrics.val\recall metrics.tn metrics.auc metrics.val\loss tags.mlflow.log-model.history tags.mlflow.source.type tags.mlflow.user tags.mlflow.project.entryPoint tags.mlflow.source.git.repoURL tags.mlflow.source.name tags.mlflow.source.git.commit tags.mlflow.project.backend tags.mlflow.gitRepoURL tags.mlflow.project.env
0 1aefda1366f04f5da5d1fc2241ad9208 0 FINISHED ./data/mlruns/0/1aefda1366f04f5da5d1fc224 2020-05-31 19:39:51.905000+00:00 2020-06-01 08:07:51.011000+00:00 0.801759 24565164.0 0.978632 0.350498 [{"run\id": "1aefda1366f04f5da5d1fc2241ad9208"… PROJECT ye53nis main https://github.com/aseltmann/fluotracify file:///beegfs/ye53nis/drmed-git 2225d6fa18cca9044960b6e86a56ec9fb4362d5d local https://github.com/aseltmann/fluotracify conda

1 rows × 52 columns

client = mlflow.tracking.MlflowClient(tracking_uri=uri)
run_idx = 0

# mlflow.entities.Experiment
print('client.get_experiment()\n', exp.to_proto())
# mlflow.entities.Metric
# metent = client.get_metric_history(run_id=runs.iloc[run_idx].run_id, key='loss')
# for i in metent:
    # print(i)

print('- - - mlflow.entities.Run - - -')
run = client.get_run(runs.iloc[run_idx].run_id)
print('run.info.to_proto\n', run.info.to_proto())
pprint.pprint(run.data.metrics, sort_dicts=False)

model_path = client.download_artifacts(
    run_id=runs.iloc[run_idx].run_id,
    path='model')
tensorboard_path = client.download_artifacts(
    run_id=runs.iloc[run_idx].run_id,
    path='tensorboard_logs')



print('\nmodel_path\n', model_path)
print('\nMLmodel file')
%cat $model_path/MLmodel
print('\nconda.yaml')
%cat $model_path/conda.yaml
print('\nkeras_module.txt')
%cat $model_path/data/keras_module.txt
client.get_experiment()
 experiment_id: "0"
name: "exp-310520-unet"
artifact_location: "file:./data/mlruns/0"
lifecycle_stage: "active"

- - - mlflow.entities.Run - - -
run.info.to_proto
 run_uuid: "1aefda1366f04f5da5d1fc2241ad9208"
experiment_id: "0"
user_id: "ye53nis"
status: FINISHED
start_time: 1590953991905
end_time: 1590998871011
artifact_uri: "file:./data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts"
lifecycle_stage: "active"
run_id: "1aefda1366f04f5da5d1fc2241ad9208"

{'learning rate': 1e-05,
 'val_loss': 0.3504984378814697,
 'precision': 0.9042773246765137,
 'fp': 711970.0,
 'loss': 0.2996121346950531,
 'val_tp': 3050870.0,
 'recall': 0.8978734016418457,
 'accuracy': 0.9549258947372437,
 'val_precision': 0.9677141904830933,
 'val_recall': 0.8017593622207642,
 'lr': 1e-05,
 'fn': 765019.0,
 'auc': 0.9786317348480225,
 'val_fp': 101786.0,
 'val_accuracy': 0.947745680809021,
 'val_tn': 12476995.0,
 'val_auc': 0.977130651473999,
 'val_fn': 754349.0,
 'tn': 24565164.0,
 'tp': 6725869.0}

model_path
 /home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/model

MLmodel file
artifact_path: model
flavors:
  keras:
    data: data
    keras_module: tensorflow.keras
    keras_version: 2.2.4-tf
  python_function:
    data: data
    env: conda.yaml
    loader_module: mlflow.keras
    python_version: 3.8.3
run_id: 1aefda1366f04f5da5d1fc2241ad9208
utc_time_created: '2020-06-01 08:05:46.358978'

conda.yaml
channels:
- defaults
dependencies:
- python=3.8.3
- pip
- pip:
  - mlflow
  - tensorflow==2.3.0-dev20200527
name: mlflow-env

keras_module.txt
tensorflow.keras

summary_path
 /home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/model_summary.txt

tensorboard_path
 /home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/tensorboard_logs
summary_path = client.download_artifacts(
    run_id=runs.iloc[run_idx].run_id,
    path='model_summary.txt')

print('summary_path\n', summary_path, '\n')
%cat $summary_path
summary_path
 /home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/model_summary.txt

Model: "functional_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_1 (InputLayer)            [(None, 16384, 1)]   0
__________________________________________________________________________________________________
encode0 (Sequential)            (None, 16384, 64)    13120       input_1[0][0]
__________________________________________________________________________________________________
mp_encode0 (MaxPooling1D)       (None, 8192, 64)     0           encode0[0][0]
__________________________________________________________________________________________________
encode1 (Sequential)            (None, 8192, 128)    75008       mp_encode0[0][0]
__________________________________________________________________________________________________
mp_encode1 (MaxPooling1D)       (None, 4096, 128)    0           encode1[0][0]
__________________________________________________________________________________________________
encode2 (Sequential)            (None, 4096, 256)    297472      mp_encode1[0][0]
__________________________________________________________________________________________________
mp_encode2 (MaxPooling1D)       (None, 2048, 256)    0           encode2[0][0]
__________________________________________________________________________________________________
encode3 (Sequential)            (None, 2048, 512)    1184768     mp_encode2[0][0]
__________________________________________________________________________________________________
mp_encode3 (MaxPooling1D)       (None, 1024, 512)    0           encode3[0][0]
__________________________________________________________________________________________________
encode4 (Sequential)            (None, 1024, 512)    1577984     mp_encode3[0][0]
__________________________________________________________________________________________________
mp_encode4 (MaxPooling1D)       (None, 512, 512)     0           encode4[0][0]
__________________________________________________________________________________________________
encode5 (Sequential)            (None, 512, 512)     1577984     mp_encode4[0][0]
__________________________________________________________________________________________________
mp_encode5 (MaxPooling1D)       (None, 256, 512)     0           encode5[0][0]
__________________________________________________________________________________________________
encode6 (Sequential)            (None, 256, 512)     1577984     mp_encode5[0][0]
__________________________________________________________________________________________________
mp_encode6 (MaxPooling1D)       (None, 128, 512)     0           encode6[0][0]
__________________________________________________________________________________________________
encode7 (Sequential)            (None, 128, 512)     1577984     mp_encode6[0][0]
__________________________________________________________________________________________________
mp_encode7 (MaxPooling1D)       (None, 64, 512)      0           encode7[0][0]
__________________________________________________________________________________________________
encode8 (Sequential)            (None, 64, 512)      1577984     mp_encode7[0][0]
__________________________________________________________________________________________________
mp_encode8 (MaxPooling1D)       (None, 32, 512)      0           encode8[0][0]
__________________________________________________________________________________________________
two_conv_center (Sequential)    (None, 32, 1024)     4728832     mp_encode8[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder8 (Sequen (None, 64, 512)      1051136     two_conv_center[0][0]
__________________________________________________________________________________________________
decoder8 (Concatenate)          (None, 64, 1024)     0           encode8[0][0]
                                                                 conv_transpose_decoder8[0][0]
__________________________________________________________________________________________________
two_conv_decoder8 (Sequential)  (None, 64, 512)      2364416     decoder8[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder7 (Sequen (None, 128, 512)     526848      two_conv_decoder8[0][0]
__________________________________________________________________________________________________
decoder7 (Concatenate)          (None, 128, 1024)    0           encode7[0][0]
                                                                 conv_transpose_decoder7[0][0]
__________________________________________________________________________________________________
two_conv_decoder7 (Sequential)  (None, 128, 512)     2364416     decoder7[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder6 (Sequen (None, 256, 512)     526848      two_conv_decoder7[0][0]
__________________________________________________________________________________________________
decoder6 (Concatenate)          (None, 256, 1024)    0           encode6[0][0]
                                                                 conv_transpose_decoder6[0][0]
__________________________________________________________________________________________________
two_conv_decoder6 (Sequential)  (None, 256, 512)     2364416     decoder6[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder5 (Sequen (None, 512, 512)     526848      two_conv_decoder6[0][0]
__________________________________________________________________________________________________
decoder5 (Concatenate)          (None, 512, 1024)    0           encode5[0][0]
                                                                 conv_transpose_decoder5[0][0]
__________________________________________________________________________________________________
two_conv_decoder5 (Sequential)  (None, 512, 512)     2364416     decoder5[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder4 (Sequen (None, 1024, 512)    526848      two_conv_decoder5[0][0]
__________________________________________________________________________________________________
decoder4 (Concatenate)          (None, 1024, 1024)   0           encode4[0][0]
                                                                 conv_transpose_decoder4[0][0]
__________________________________________________________________________________________________
two_conv_decoder4 (Sequential)  (None, 1024, 512)    2364416     decoder4[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder3 (Sequen (None, 2048, 512)    526848      two_conv_decoder4[0][0]
__________________________________________________________________________________________________
decoder3 (Concatenate)          (None, 2048, 1024)   0           encode3[0][0]
                                                                 conv_transpose_decoder3[0][0]
__________________________________________________________________________________________________
two_conv_decoder3 (Sequential)  (None, 2048, 512)    2364416     decoder3[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder2 (Sequen (None, 4096, 256)    263424      two_conv_decoder3[0][0]
__________________________________________________________________________________________________
decoder2 (Concatenate)          (None, 4096, 512)    0           encode2[0][0]
                                                                 conv_transpose_decoder2[0][0]
__________________________________________________________________________________________________
two_conv_decoder2 (Sequential)  (None, 4096, 256)    592384      decoder2[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder1 (Sequen (None, 8192, 128)    66176       two_conv_decoder2[0][0]
__________________________________________________________________________________________________
decoder1 (Concatenate)          (None, 8192, 256)    0           encode1[0][0]
                                                                 conv_transpose_decoder1[0][0]
__________________________________________________________________________________________________
two_conv_decoder1 (Sequential)  (None, 8192, 128)    148736      decoder1[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder0 (Sequen (None, 16384, 64)    16704       two_conv_decoder1[0][0]
__________________________________________________________________________________________________
decoder0 (Concatenate)          (None, 16384, 128)   0           encode0[0][0]
                                                                 conv_transpose_decoder0[0][0]
__________________________________________________________________________________________________
two_conv_decoder0 (Sequential)  (None, 16384, 64)    37504       decoder0[0][0]
__________________________________________________________________________________________________
conv1d_38 (Conv1D)              (None, 16384, 1)     65          two_conv_decoder0[0][0]
==================================================================================================
Total params: 33,185,985
Trainable params: 33,146,689
Non-trainable params: 39,296
__________________________________________________________________________________________________
  print('\ntensorboard_path\n', tensorboard_path)
  !ls $tensorboard_path

# https://stackoverflow.com/questions/41074688/how-do-you-read-tensorboard-files-programmatically
from tensorboard.backend.event_processing import event_accumulator
path = str(tensorboard_path) + '/train'
path2 = str()
print(path)
ea = event_accumulator.EventAccumulator(path=path,
    size_guidance={ # see below regarding this argument
        event_accumulator.COMPRESSED_HISTOGRAMS: 500,
        event_accumulator.IMAGES: 4,
        event_accumulator.AUDIO: 4,
        event_accumulator.SCALARS: 0,
        event_accumulator.HISTOGRAMS: 1,
    })

ea.Reload() # loads events from file
tensorboard_path
 /home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/tensorboard_logs
image  metrics	train  validation
/home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/tensorboard_logs/train
<tensorboard.backend.event_processing.event_accumulator.EventAccumulator at 0x7fdb70770c70>
ea.Tags()
{'images': ['encode0/conv1d/kernel_0/image/0',
    'encode0/conv1d/bias_0/image/0',
    'encode0/batch_normalization/gamma_0/image/0',
    'encode0/batch_normalization/beta_0/image/0',
    'encode0/batch_normalization/moving_mean_0/image/0',
    'encode0/batch_normalization/moving_variance_0/image/0',
    'encode0/conv1d_1/kernel_0/image/0',
    'encode0/conv1d_1/kernel_0/image/1',
    'encode0/conv1d_1/kernel_0/image/2',
    'encode0/conv1d_1/bias_0/image/0',
    'encode0/batch_normalization_1/gamma_0/image/0',
    'encode0/batch_normalization_1/beta_0/image/0',
    'encode0/batch_normalization_1/moving_mean_0/image/0',
    'encode0/batch_normalization_1/moving_variance_0/image/0',
    'encode1/conv1d_2/kernel_0/image/0',
    'encode1/conv1d_2/kernel_0/image/1',
    'encode1/conv1d_2/kernel_0/image/2',
    'encode1/conv1d_2/bias_0/image/0',
    'encode1/batch_normalization_2/gamma_0/image/0',
    'encode1/batch_normalization_2/beta_0/image/0',
    'encode1/batch_normalization_2/moving_mean_0/image/0',
    'encode1/batch_normalization_2/moving_variance_0/image/0',
    'encode1/conv1d_3/kernel_0/image/0',
    'encode1/conv1d_3/kernel_0/image/1',
    'encode1/conv1d_3/kernel_0/image/2',
    'encode1/conv1d_3/bias_0/image/0',
    'encode1/batch_normalization_3/gamma_0/image/0',
    'encode1/batch_normalization_3/beta_0/image/0',
    'encode1/batch_normalization_3/moving_mean_0/image/0',
    'encode1/batch_normalization_3/moving_variance_0/image/0',
    'encode2/conv1d_4/kernel_0/image/0',
    'encode2/conv1d_4/kernel_0/image/1',
    'encode2/conv1d_4/kernel_0/image/2',
    'encode2/conv1d_4/bias_0/image/0',
    'encode2/batch_normalization_4/gamma_0/image/0',
    'encode2/batch_normalization_4/beta_0/image/0',
    'encode2/batch_normalization_4/moving_mean_0/image/0',
    'encode2/batch_normalization_4/moving_variance_0/image/0',
    'encode2/conv1d_5/kernel_0/image/0',
    'encode2/conv1d_5/kernel_0/image/1',
    'encode2/conv1d_5/kernel_0/image/2',
    'encode2/conv1d_5/bias_0/image/0',
    'encode2/batch_normalization_5/gamma_0/image/0',
    'encode2/batch_normalization_5/beta_0/image/0',
    'encode2/batch_normalization_5/moving_mean_0/image/0',
    'encode2/batch_normalization_5/moving_variance_0/image/0',
    'encode3/conv1d_6/kernel_0/image/0',
    'encode3/conv1d_6/kernel_0/image/1',
    'encode3/conv1d_6/kernel_0/image/2',
    'encode3/conv1d_6/bias_0/image/0',
    'encode3/batch_normalization_6/gamma_0/image/0',
    'encode3/batch_normalization_6/beta_0/image/0',
    'encode3/batch_normalization_6/moving_mean_0/image/0',
    'encode3/batch_normalization_6/moving_variance_0/image/0',
    'encode3/conv1d_7/kernel_0/image/0',
    'encode3/conv1d_7/kernel_0/image/1',
    'encode3/conv1d_7/kernel_0/image/2',
    'encode3/conv1d_7/bias_0/image/0',
    'encode3/batch_normalization_7/gamma_0/image/0',
    'encode3/batch_normalization_7/beta_0/image/0',
    'encode3/batch_normalization_7/moving_mean_0/image/0',
    'encode3/batch_normalization_7/moving_variance_0/image/0',
    'encode4/conv1d_8/kernel_0/image/0',
    'encode4/conv1d_8/kernel_0/image/1',
    'encode4/conv1d_8/kernel_0/image/2',
    'encode4/conv1d_8/bias_0/image/0',
    'encode4/batch_normalization_8/gamma_0/image/0',
    'encode4/batch_normalization_8/beta_0/image/0',
    'encode4/batch_normalization_8/moving_mean_0/image/0',
    'encode4/batch_normalization_8/moving_variance_0/image/0',
    'encode4/conv1d_9/kernel_0/image/0',
    'encode4/conv1d_9/kernel_0/image/1',
    'encode4/conv1d_9/kernel_0/image/2',
    'encode4/conv1d_9/bias_0/image/0',
    'encode4/batch_normalization_9/gamma_0/image/0',
    'encode4/batch_normalization_9/beta_0/image/0',
    'encode4/batch_normalization_9/moving_mean_0/image/0',
    'encode4/batch_normalization_9/moving_variance_0/image/0',
    'encode5/conv1d_10/kernel_0/image/0',
    'encode5/conv1d_10/kernel_0/image/1',
    'encode5/conv1d_10/kernel_0/image/2',
    'encode5/conv1d_10/bias_0/image/0',
    'encode5/batch_normalization_10/gamma_0/image/0',
    'encode5/batch_normalization_10/beta_0/image/0',
    'encode5/batch_normalization_10/moving_mean_0/image/0',
    'encode5/batch_normalization_10/moving_variance_0/image/0',
    'encode5/conv1d_11/kernel_0/image/0',
    'encode5/conv1d_11/kernel_0/image/1',
    'encode5/conv1d_11/kernel_0/image/2',
    'encode5/conv1d_11/bias_0/image/0',
    'encode5/batch_normalization_11/gamma_0/image/0',
    'encode5/batch_normalization_11/beta_0/image/0',
    'encode5/batch_normalization_11/moving_mean_0/image/0',
    'encode5/batch_normalization_11/moving_variance_0/image/0',
    'encode6/conv1d_12/kernel_0/image/0',
    'encode6/conv1d_12/kernel_0/image/1',
    'encode6/conv1d_12/kernel_0/image/2',
    'encode6/conv1d_12/bias_0/image/0',
    'encode6/batch_normalization_12/gamma_0/image/0',
    'encode6/batch_normalization_12/beta_0/image/0',
    'encode6/batch_normalization_12/moving_mean_0/image/0',
    'encode6/batch_normalization_12/moving_variance_0/image/0',
    'encode6/conv1d_13/kernel_0/image/0',
    'encode6/conv1d_13/kernel_0/image/1',
    'encode6/conv1d_13/kernel_0/image/2',
    'encode6/conv1d_13/bias_0/image/0',
    'encode6/batch_normalization_13/gamma_0/image/0',
    'encode6/batch_normalization_13/beta_0/image/0',
    'encode6/batch_normalization_13/moving_mean_0/image/0',
    'encode6/batch_normalization_13/moving_variance_0/image/0',
    'encode7/conv1d_14/kernel_0/image/0',
    'encode7/conv1d_14/kernel_0/image/1',
    'encode7/conv1d_14/kernel_0/image/2',
    'encode7/conv1d_14/bias_0/image/0',
    'encode7/batch_normalization_14/gamma_0/image/0',
    'encode7/batch_normalization_14/beta_0/image/0',
    'encode7/batch_normalization_14/moving_mean_0/image/0',
    'encode7/batch_normalization_14/moving_variance_0/image/0',
    'encode7/conv1d_15/kernel_0/image/0',
    'encode7/conv1d_15/kernel_0/image/1',
    'encode7/conv1d_15/kernel_0/image/2',
    'encode7/conv1d_15/bias_0/image/0',
    'encode7/batch_normalization_15/gamma_0/image/0',
    'encode7/batch_normalization_15/beta_0/image/0',
    'encode7/batch_normalization_15/moving_mean_0/image/0',
    'encode7/batch_normalization_15/moving_variance_0/image/0',
    'encode8/conv1d_16/kernel_0/image/0',
    'encode8/conv1d_16/kernel_0/image/1',
    'encode8/conv1d_16/kernel_0/image/2',
    'encode8/conv1d_16/bias_0/image/0',
    'encode8/batch_normalization_16/gamma_0/image/0',
    'encode8/batch_normalization_16/beta_0/image/0',
    'encode8/batch_normalization_16/moving_mean_0/image/0',
    'encode8/batch_normalization_16/moving_variance_0/image/0',
    'encode8/conv1d_17/kernel_0/image/0',
    'encode8/conv1d_17/kernel_0/image/1',
    'encode8/conv1d_17/kernel_0/image/2',
    'encode8/conv1d_17/bias_0/image/0',
    'encode8/batch_normalization_17/gamma_0/image/0',
    'encode8/batch_normalization_17/beta_0/image/0',
    'encode8/batch_normalization_17/moving_mean_0/image/0',
    'encode8/batch_normalization_17/moving_variance_0/image/0',
    'two_conv_center/conv1d_18/kernel_0/image/0',
    'two_conv_center/conv1d_18/kernel_0/image/1',
    'two_conv_center/conv1d_18/kernel_0/image/2',
    'two_conv_center/conv1d_18/bias_0/image/0',
    'two_conv_center/batch_normalization_18/gamma_0/image/0',
    'two_conv_center/batch_normalization_18/beta_0/image/0',
    'two_conv_center/batch_normalization_18/moving_mean_0/image/0',
    'two_conv_center/batch_normalization_18/moving_variance_0/image/0',
    'two_conv_center/conv1d_19/kernel_0/image/0',
    'two_conv_center/conv1d_19/kernel_0/image/1',
    'two_conv_center/conv1d_19/kernel_0/image/2',
    'two_conv_center/conv1d_19/bias_0/image/0',
    'two_conv_center/batch_normalization_19/gamma_0/image/0',
    'two_conv_center/batch_normalization_19/beta_0/image/0',
    'two_conv_center/batch_normalization_19/moving_mean_0/image/0',
    'two_conv_center/batch_normalization_19/moving_variance_0/image/0',
    'conv_transpose_decoder8/conv1d_transpose/kernel_0/image/0',
    'conv_transpose_decoder8/conv1d_transpose/kernel_0/image/1',
    'conv_transpose_decoder8/conv1d_transpose/kernel_0/image/2',
    'conv_transpose_decoder8/conv1d_transpose/bias_0/image/0',
    'conv_transpose_decoder8/batch_normalization_20/gamma_0/image/0',
    'conv_transpose_decoder8/batch_normalization_20/beta_0/image/0',
    'conv_transpose_decoder8/batch_normalization_20/moving_mean_0/image/0',
    'conv_transpose_decoder8/batch_normalization_20/moving_variance_0/image/0',
    'two_conv_decoder8/conv1d_20/kernel_0/image/0',
    'two_conv_decoder8/conv1d_20/kernel_0/image/1',
    'two_conv_decoder8/conv1d_20/kernel_0/image/2',
    'two_conv_decoder8/conv1d_20/bias_0/image/0',
    'two_conv_decoder8/batch_normalization_21/gamma_0/image/0',
    'two_conv_decoder8/batch_normalization_21/beta_0/image/0',
    'two_conv_decoder8/batch_normalization_21/moving_mean_0/image/0',
    'two_conv_decoder8/batch_normalization_21/moving_variance_0/image/0',
    'two_conv_decoder8/conv1d_21/kernel_0/image/0',
    'two_conv_decoder8/conv1d_21/kernel_0/image/1',
    'two_conv_decoder8/conv1d_21/kernel_0/image/2',
    'two_conv_decoder8/conv1d_21/bias_0/image/0',
    'two_conv_decoder8/batch_normalization_22/gamma_0/image/0',
    'two_conv_decoder8/batch_normalization_22/beta_0/image/0',
    'two_conv_decoder8/batch_normalization_22/moving_mean_0/image/0',
    'two_conv_decoder8/batch_normalization_22/moving_variance_0/image/0',
    'conv_transpose_decoder7/conv1d_transpose_1/kernel_0/image/0',
    'conv_transpose_decoder7/conv1d_transpose_1/kernel_0/image/1',
    'conv_transpose_decoder7/conv1d_transpose_1/kernel_0/image/2',
    'conv_transpose_decoder7/conv1d_transpose_1/bias_0/image/0',
    'conv_transpose_decoder7/batch_normalization_23/gamma_0/image/0',
    'conv_transpose_decoder7/batch_normalization_23/beta_0/image/0',
    'conv_transpose_decoder7/batch_normalization_23/moving_mean_0/image/0',
    'conv_transpose_decoder7/batch_normalization_23/moving_variance_0/image/0',
    'two_conv_decoder7/conv1d_22/kernel_0/image/0',
    'two_conv_decoder7/conv1d_22/kernel_0/image/1',
    'two_conv_decoder7/conv1d_22/kernel_0/image/2',
    'two_conv_decoder7/conv1d_22/bias_0/image/0',
    'two_conv_decoder7/batch_normalization_24/gamma_0/image/0',
    'two_conv_decoder7/batch_normalization_24/beta_0/image/0',
    'two_conv_decoder7/batch_normalization_24/moving_mean_0/image/0',
    'two_conv_decoder7/batch_normalization_24/moving_variance_0/image/0',
    'two_conv_decoder7/conv1d_23/kernel_0/image/0',
    'two_conv_decoder7/conv1d_23/kernel_0/image/1',
    'two_conv_decoder7/conv1d_23/kernel_0/image/2',
    'two_conv_decoder7/conv1d_23/bias_0/image/0',
    'two_conv_decoder7/batch_normalization_25/gamma_0/image/0',
    'two_conv_decoder7/batch_normalization_25/beta_0/image/0',
    'two_conv_decoder7/batch_normalization_25/moving_mean_0/image/0',
    'two_conv_decoder7/batch_normalization_25/moving_variance_0/image/0',
    'conv_transpose_decoder6/conv1d_transpose_2/kernel_0/image/0',
    'conv_transpose_decoder6/conv1d_transpose_2/kernel_0/image/1',
    'conv_transpose_decoder6/conv1d_transpose_2/kernel_0/image/2',
    'conv_transpose_decoder6/conv1d_transpose_2/bias_0/image/0',
    'conv_transpose_decoder6/batch_normalization_26/gamma_0/image/0',
    'conv_transpose_decoder6/batch_normalization_26/beta_0/image/0',
    'conv_transpose_decoder6/batch_normalization_26/moving_mean_0/image/0',
    'conv_transpose_decoder6/batch_normalization_26/moving_variance_0/image/0',
    'two_conv_decoder6/conv1d_24/kernel_0/image/0',
    'two_conv_decoder6/conv1d_24/kernel_0/image/1',
    'two_conv_decoder6/conv1d_24/kernel_0/image/2',
    'two_conv_decoder6/conv1d_24/bias_0/image/0',
    'two_conv_decoder6/batch_normalization_27/gamma_0/image/0',
    'two_conv_decoder6/batch_normalization_27/beta_0/image/0',
    'two_conv_decoder6/batch_normalization_27/moving_mean_0/image/0',
    'two_conv_decoder6/batch_normalization_27/moving_variance_0/image/0',
    'two_conv_decoder6/conv1d_25/kernel_0/image/0',
    'two_conv_decoder6/conv1d_25/kernel_0/image/1',
    'two_conv_decoder6/conv1d_25/kernel_0/image/2',
    'two_conv_decoder6/conv1d_25/bias_0/image/0',
    'two_conv_decoder6/batch_normalization_28/gamma_0/image/0',
    'two_conv_decoder6/batch_normalization_28/beta_0/image/0',
    'two_conv_decoder6/batch_normalization_28/moving_mean_0/image/0',
    'two_conv_decoder6/batch_normalization_28/moving_variance_0/image/0',
    'conv_transpose_decoder5/conv1d_transpose_3/kernel_0/image/0',
    'conv_transpose_decoder5/conv1d_transpose_3/kernel_0/image/1',
    'conv_transpose_decoder5/conv1d_transpose_3/kernel_0/image/2',
    'conv_transpose_decoder5/conv1d_transpose_3/bias_0/image/0',
    'conv_transpose_decoder5/batch_normalization_29/gamma_0/image/0',
    'conv_transpose_decoder5/batch_normalization_29/beta_0/image/0',
    'conv_transpose_decoder5/batch_normalization_29/moving_mean_0/image/0',
    'conv_transpose_decoder5/batch_normalization_29/moving_variance_0/image/0',
    'two_conv_decoder5/conv1d_26/kernel_0/image/0',
    'two_conv_decoder5/conv1d_26/kernel_0/image/1',
    'two_conv_decoder5/conv1d_26/kernel_0/image/2',
    'two_conv_decoder5/conv1d_26/bias_0/image/0',
    'two_conv_decoder5/batch_normalization_30/gamma_0/image/0',
    'two_conv_decoder5/batch_normalization_30/beta_0/image/0',
    'two_conv_decoder5/batch_normalization_30/moving_mean_0/image/0',
    'two_conv_decoder5/batch_normalization_30/moving_variance_0/image/0',
    'two_conv_decoder5/conv1d_27/kernel_0/image/0',
    'two_conv_decoder5/conv1d_27/kernel_0/image/1',
    'two_conv_decoder5/conv1d_27/kernel_0/image/2',
    'two_conv_decoder5/conv1d_27/bias_0/image/0',
    'two_conv_decoder5/batch_normalization_31/gamma_0/image/0',
    'two_conv_decoder5/batch_normalization_31/beta_0/image/0',
    'two_conv_decoder5/batch_normalization_31/moving_mean_0/image/0',
    'two_conv_decoder5/batch_normalization_31/moving_variance_0/image/0',
    'conv_transpose_decoder4/conv1d_transpose_4/kernel_0/image/0',
    'conv_transpose_decoder4/conv1d_transpose_4/kernel_0/image/1',
    'conv_transpose_decoder4/conv1d_transpose_4/kernel_0/image/2',
    'conv_transpose_decoder4/conv1d_transpose_4/bias_0/image/0',
    'conv_transpose_decoder4/batch_normalization_32/gamma_0/image/0',
    'conv_transpose_decoder4/batch_normalization_32/beta_0/image/0',
    'conv_transpose_decoder4/batch_normalization_32/moving_mean_0/image/0',
    'conv_transpose_decoder4/batch_normalization_32/moving_variance_0/image/0',
    'two_conv_decoder4/conv1d_28/kernel_0/image/0',
    'two_conv_decoder4/conv1d_28/kernel_0/image/1',
    'two_conv_decoder4/conv1d_28/kernel_0/image/2',
    'two_conv_decoder4/conv1d_28/bias_0/image/0',
    'two_conv_decoder4/batch_normalization_33/gamma_0/image/0',
    'two_conv_decoder4/batch_normalization_33/beta_0/image/0',
    'two_conv_decoder4/batch_normalization_33/moving_mean_0/image/0',
    'two_conv_decoder4/batch_normalization_33/moving_variance_0/image/0',
    'two_conv_decoder4/conv1d_29/kernel_0/image/0',
    'two_conv_decoder4/conv1d_29/kernel_0/image/1',
    'two_conv_decoder4/conv1d_29/kernel_0/image/2',
    'two_conv_decoder4/conv1d_29/bias_0/image/0',
    'two_conv_decoder4/batch_normalization_34/gamma_0/image/0',
    'two_conv_decoder4/batch_normalization_34/beta_0/image/0',
    'two_conv_decoder4/batch_normalization_34/moving_mean_0/image/0',
    'two_conv_decoder4/batch_normalization_34/moving_variance_0/image/0',
    'conv_transpose_decoder3/conv1d_transpose_5/kernel_0/image/0',
    'conv_transpose_decoder3/conv1d_transpose_5/kernel_0/image/1',
    'conv_transpose_decoder3/conv1d_transpose_5/kernel_0/image/2',
    'conv_transpose_decoder3/conv1d_transpose_5/bias_0/image/0',
    'conv_transpose_decoder3/batch_normalization_35/gamma_0/image/0',
    'conv_transpose_decoder3/batch_normalization_35/beta_0/image/0',
    'conv_transpose_decoder3/batch_normalization_35/moving_mean_0/image/0',
    'conv_transpose_decoder3/batch_normalization_35/moving_variance_0/image/0',
    'two_conv_decoder3/conv1d_30/kernel_0/image/0',
    'two_conv_decoder3/conv1d_30/kernel_0/image/1',
    'two_conv_decoder3/conv1d_30/kernel_0/image/2',
    'two_conv_decoder3/conv1d_30/bias_0/image/0',
    'two_conv_decoder3/batch_normalization_36/gamma_0/image/0',
    'two_conv_decoder3/batch_normalization_36/beta_0/image/0',
    'two_conv_decoder3/batch_normalization_36/moving_mean_0/image/0',
    'two_conv_decoder3/batch_normalization_36/moving_variance_0/image/0',
    'two_conv_decoder3/conv1d_31/kernel_0/image/0',
    'two_conv_decoder3/conv1d_31/kernel_0/image/1',
    'two_conv_decoder3/conv1d_31/kernel_0/image/2',
    'two_conv_decoder3/conv1d_31/bias_0/image/0',
    'two_conv_decoder3/batch_normalization_37/gamma_0/image/0',
    'two_conv_decoder3/batch_normalization_37/beta_0/image/0',
    'two_conv_decoder3/batch_normalization_37/moving_mean_0/image/0',
    'two_conv_decoder3/batch_normalization_37/moving_variance_0/image/0',
    'conv_transpose_decoder2/conv1d_transpose_6/kernel_0/image/0',
    'conv_transpose_decoder2/conv1d_transpose_6/kernel_0/image/1',
    'conv_transpose_decoder2/conv1d_transpose_6/kernel_0/image/2',
    'conv_transpose_decoder2/conv1d_transpose_6/bias_0/image/0',
    'conv_transpose_decoder2/batch_normalization_38/gamma_0/image/0',
    'conv_transpose_decoder2/batch_normalization_38/beta_0/image/0',
    'conv_transpose_decoder2/batch_normalization_38/moving_mean_0/image/0',
    'conv_transpose_decoder2/batch_normalization_38/moving_variance_0/image/0',
    'two_conv_decoder2/conv1d_32/kernel_0/image/0',
    'two_conv_decoder2/conv1d_32/kernel_0/image/1',
    'two_conv_decoder2/conv1d_32/kernel_0/image/2',
    'two_conv_decoder2/conv1d_32/bias_0/image/0',
    'two_conv_decoder2/batch_normalization_39/gamma_0/image/0',
    'two_conv_decoder2/batch_normalization_39/beta_0/image/0',
    'two_conv_decoder2/batch_normalization_39/moving_mean_0/image/0',
    'two_conv_decoder2/batch_normalization_39/moving_variance_0/image/0',
    'two_conv_decoder2/conv1d_33/kernel_0/image/0',
    'two_conv_decoder2/conv1d_33/kernel_0/image/1',
    'two_conv_decoder2/conv1d_33/kernel_0/image/2',
    'two_conv_decoder2/conv1d_33/bias_0/image/0',
    'two_conv_decoder2/batch_normalization_40/gamma_0/image/0',
    'two_conv_decoder2/batch_normalization_40/beta_0/image/0',
    'two_conv_decoder2/batch_normalization_40/moving_mean_0/image/0',
    'two_conv_decoder2/batch_normalization_40/moving_variance_0/image/0',
    'conv_transpose_decoder1/conv1d_transpose_7/kernel_0/image/0',
    'conv_transpose_decoder1/conv1d_transpose_7/kernel_0/image/1',
    'conv_transpose_decoder1/conv1d_transpose_7/kernel_0/image/2',
    'conv_transpose_decoder1/conv1d_transpose_7/bias_0/image/0',
    'conv_transpose_decoder1/batch_normalization_41/gamma_0/image/0',
    'conv_transpose_decoder1/batch_normalization_41/beta_0/image/0',
    'conv_transpose_decoder1/batch_normalization_41/moving_mean_0/image/0',
    'conv_transpose_decoder1/batch_normalization_41/moving_variance_0/image/0',
    'two_conv_decoder1/conv1d_34/kernel_0/image/0',
    'two_conv_decoder1/conv1d_34/kernel_0/image/1',
    'two_conv_decoder1/conv1d_34/kernel_0/image/2',
    'two_conv_decoder1/conv1d_34/bias_0/image/0',
    'two_conv_decoder1/batch_normalization_42/gamma_0/image/0',
    'two_conv_decoder1/batch_normalization_42/beta_0/image/0',
    'two_conv_decoder1/batch_normalization_42/moving_mean_0/image/0',
    'two_conv_decoder1/batch_normalization_42/moving_variance_0/image/0',
    'two_conv_decoder1/conv1d_35/kernel_0/image/0',
    'two_conv_decoder1/conv1d_35/kernel_0/image/1',
    'two_conv_decoder1/conv1d_35/kernel_0/image/2',
    'two_conv_decoder1/conv1d_35/bias_0/image/0',
    'two_conv_decoder1/batch_normalization_43/gamma_0/image/0',
    'two_conv_decoder1/batch_normalization_43/beta_0/image/0',
    'two_conv_decoder1/batch_normalization_43/moving_mean_0/image/0',
    'two_conv_decoder1/batch_normalization_43/moving_variance_0/image/0',
    'conv_transpose_decoder0/conv1d_transpose_8/kernel_0/image/0',
    'conv_transpose_decoder0/conv1d_transpose_8/kernel_0/image/1',
    'conv_transpose_decoder0/conv1d_transpose_8/kernel_0/image/2',
    'conv_transpose_decoder0/conv1d_transpose_8/bias_0/image/0',
    'conv_transpose_decoder0/batch_normalization_44/gamma_0/image/0',
    'conv_transpose_decoder0/batch_normalization_44/beta_0/image/0',
    'conv_transpose_decoder0/batch_normalization_44/moving_mean_0/image/0',
    'conv_transpose_decoder0/batch_normalization_44/moving_variance_0/image/0',
    'two_conv_decoder0/conv1d_36/kernel_0/image/0',
    'two_conv_decoder0/conv1d_36/kernel_0/image/1',
    'two_conv_decoder0/conv1d_36/kernel_0/image/2',
    'two_conv_decoder0/conv1d_36/bias_0/image/0',
    'two_conv_decoder0/batch_normalization_45/gamma_0/image/0',
    'two_conv_decoder0/batch_normalization_45/beta_0/image/0',
    'two_conv_decoder0/batch_normalization_45/moving_mean_0/image/0',
    'two_conv_decoder0/batch_normalization_45/moving_variance_0/image/0',
    'two_conv_decoder0/conv1d_37/kernel_0/image/0',
    'two_conv_decoder0/conv1d_37/kernel_0/image/1',
    'two_conv_decoder0/conv1d_37/kernel_0/image/2',
    'two_conv_decoder0/conv1d_37/bias_0/image/0',
    'two_conv_decoder0/batch_normalization_46/gamma_0/image/0',
    'two_conv_decoder0/batch_normalization_46/beta_0/image/0',
    'two_conv_decoder0/batch_normalization_46/moving_mean_0/image/0',
    'two_conv_decoder0/batch_normalization_46/moving_variance_0/image/0',
    'conv1d_38/kernel_0/image/0'],
   'audio': [],
   'histograms': ['encode0/conv1d/kernel_0',
    'encode0/conv1d/bias_0',
    'encode0/batch_normalization/gamma_0',
    'encode0/batch_normalization/beta_0',
    'encode0/batch_normalization/moving_mean_0',
    'encode0/batch_normalization/moving_variance_0',
    'encode0/conv1d_1/kernel_0',
    'encode0/conv1d_1/bias_0',
    'encode0/batch_normalization_1/gamma_0',
    'encode0/batch_normalization_1/beta_0',
    'encode0/batch_normalization_1/moving_mean_0',
    'encode0/batch_normalization_1/moving_variance_0',
    'encode1/conv1d_2/kernel_0',
    'encode1/conv1d_2/bias_0',
    'encode1/batch_normalization_2/gamma_0',
    'encode1/batch_normalization_2/beta_0',
    'encode1/batch_normalization_2/moving_mean_0',
    'encode1/batch_normalization_2/moving_variance_0',
    'encode1/conv1d_3/kernel_0',
    'encode1/conv1d_3/bias_0',
    'encode1/batch_normalization_3/gamma_0',
    'encode1/batch_normalization_3/beta_0',
    'encode1/batch_normalization_3/moving_mean_0',
    'encode1/batch_normalization_3/moving_variance_0',
    'encode2/conv1d_4/kernel_0',
    'encode2/conv1d_4/bias_0',
    'encode2/batch_normalization_4/gamma_0',
    'encode2/batch_normalization_4/beta_0',
    'encode2/batch_normalization_4/moving_mean_0',
    'encode2/batch_normalization_4/moving_variance_0',
    'encode2/conv1d_5/kernel_0',
    'encode2/conv1d_5/bias_0',
    'encode2/batch_normalization_5/gamma_0',
    'encode2/batch_normalization_5/beta_0',
    'encode2/batch_normalization_5/moving_mean_0',
    'encode2/batch_normalization_5/moving_variance_0',
    'encode3/conv1d_6/kernel_0',
    'encode3/conv1d_6/bias_0',
    'encode3/batch_normalization_6/gamma_0',
    'encode3/batch_normalization_6/beta_0',
    'encode3/batch_normalization_6/moving_mean_0',
    'encode3/batch_normalization_6/moving_variance_0',
    'encode3/conv1d_7/kernel_0',
    'encode3/conv1d_7/bias_0',
    'encode3/batch_normalization_7/gamma_0',
    'encode3/batch_normalization_7/beta_0',
    'encode3/batch_normalization_7/moving_mean_0',
    'encode3/batch_normalization_7/moving_variance_0',
    'encode4/conv1d_8/kernel_0',
    'encode4/conv1d_8/bias_0',
    'encode4/batch_normalization_8/gamma_0',
    'encode4/batch_normalization_8/beta_0',
    'encode4/batch_normalization_8/moving_mean_0',
    'encode4/batch_normalization_8/moving_variance_0',
    'encode4/conv1d_9/kernel_0',
    'encode4/conv1d_9/bias_0',
    'encode4/batch_normalization_9/gamma_0',
    'encode4/batch_normalization_9/beta_0',
    'encode4/batch_normalization_9/moving_mean_0',
    'encode4/batch_normalization_9/moving_variance_0',
    'encode5/conv1d_10/kernel_0',
    'encode5/conv1d_10/bias_0',
    'encode5/batch_normalization_10/gamma_0',
    'encode5/batch_normalization_10/beta_0',
    'encode5/batch_normalization_10/moving_mean_0',
    'encode5/batch_normalization_10/moving_variance_0',
    'encode5/conv1d_11/kernel_0',
    'encode5/conv1d_11/bias_0',
    'encode5/batch_normalization_11/gamma_0',
    'encode5/batch_normalization_11/beta_0',
    'encode5/batch_normalization_11/moving_mean_0',
    'encode5/batch_normalization_11/moving_variance_0',
    'encode6/conv1d_12/kernel_0',
    'encode6/conv1d_12/bias_0',
    'encode6/batch_normalization_12/gamma_0',
    'encode6/batch_normalization_12/beta_0',
    'encode6/batch_normalization_12/moving_mean_0',
    'encode6/batch_normalization_12/moving_variance_0',
    'encode6/conv1d_13/kernel_0',
    'encode6/conv1d_13/bias_0',
    'encode6/batch_normalization_13/gamma_0',
    'encode6/batch_normalization_13/beta_0',
    'encode6/batch_normalization_13/moving_mean_0',
    'encode6/batch_normalization_13/moving_variance_0',
    'encode7/conv1d_14/kernel_0',
    'encode7/conv1d_14/bias_0',
    'encode7/batch_normalization_14/gamma_0',
    'encode7/batch_normalization_14/beta_0',
    'encode7/batch_normalization_14/moving_mean_0',
    'encode7/batch_normalization_14/moving_variance_0',
    'encode7/conv1d_15/kernel_0',
    'encode7/conv1d_15/bias_0',
    'encode7/batch_normalization_15/gamma_0',
    'encode7/batch_normalization_15/beta_0',
    'encode7/batch_normalization_15/moving_mean_0',
    'encode7/batch_normalization_15/moving_variance_0',
    'encode8/conv1d_16/kernel_0',
    'encode8/conv1d_16/bias_0',
    'encode8/batch_normalization_16/gamma_0',
    'encode8/batch_normalization_16/beta_0',
    'encode8/batch_normalization_16/moving_mean_0',
    'encode8/batch_normalization_16/moving_variance_0',
    'encode8/conv1d_17/kernel_0',
    'encode8/conv1d_17/bias_0',
    'encode8/batch_normalization_17/gamma_0',
    'encode8/batch_normalization_17/beta_0',
    'encode8/batch_normalization_17/moving_mean_0',
    'encode8/batch_normalization_17/moving_variance_0',
    'two_conv_center/conv1d_18/kernel_0',
    'two_conv_center/conv1d_18/bias_0',
    'two_conv_center/batch_normalization_18/gamma_0',
    'two_conv_center/batch_normalization_18/beta_0',
    'two_conv_center/batch_normalization_18/moving_mean_0',
    'two_conv_center/batch_normalization_18/moving_variance_0',
    'two_conv_center/conv1d_19/kernel_0',
    'two_conv_center/conv1d_19/bias_0',
    'two_conv_center/batch_normalization_19/gamma_0',
    'two_conv_center/batch_normalization_19/beta_0',
    'two_conv_center/batch_normalization_19/moving_mean_0',
    'two_conv_center/batch_normalization_19/moving_variance_0',
    'conv_transpose_decoder8/conv1d_transpose/kernel_0',
    'conv_transpose_decoder8/conv1d_transpose/bias_0',
    'conv_transpose_decoder8/batch_normalization_20/gamma_0',
    'conv_transpose_decoder8/batch_normalization_20/beta_0',
    'conv_transpose_decoder8/batch_normalization_20/moving_mean_0',
    'conv_transpose_decoder8/batch_normalization_20/moving_variance_0',
    'two_conv_decoder8/conv1d_20/kernel_0',
    'two_conv_decoder8/conv1d_20/bias_0',
    'two_conv_decoder8/batch_normalization_21/gamma_0',
    'two_conv_decoder8/batch_normalization_21/beta_0',
    'two_conv_decoder8/batch_normalization_21/moving_mean_0',
    'two_conv_decoder8/batch_normalization_21/moving_variance_0',
    'two_conv_decoder8/conv1d_21/kernel_0',
    'two_conv_decoder8/conv1d_21/bias_0',
    'two_conv_decoder8/batch_normalization_22/gamma_0',
    'two_conv_decoder8/batch_normalization_22/beta_0',
    'two_conv_decoder8/batch_normalization_22/moving_mean_0',
    'two_conv_decoder8/batch_normalization_22/moving_variance_0',
    'conv_transpose_decoder7/conv1d_transpose_1/kernel_0',
    'conv_transpose_decoder7/conv1d_transpose_1/bias_0',
    'conv_transpose_decoder7/batch_normalization_23/gamma_0',
    'conv_transpose_decoder7/batch_normalization_23/beta_0',
    'conv_transpose_decoder7/batch_normalization_23/moving_mean_0',
    'conv_transpose_decoder7/batch_normalization_23/moving_variance_0',
    'two_conv_decoder7/conv1d_22/kernel_0',
    'two_conv_decoder7/conv1d_22/bias_0',
    'two_conv_decoder7/batch_normalization_24/gamma_0',
    'two_conv_decoder7/batch_normalization_24/beta_0',
    'two_conv_decoder7/batch_normalization_24/moving_mean_0',
    'two_conv_decoder7/batch_normalization_24/moving_variance_0',
    'two_conv_decoder7/conv1d_23/kernel_0',
    'two_conv_decoder7/conv1d_23/bias_0',
    'two_conv_decoder7/batch_normalization_25/gamma_0',
    'two_conv_decoder7/batch_normalization_25/beta_0',
    'two_conv_decoder7/batch_normalization_25/moving_mean_0',
    'two_conv_decoder7/batch_normalization_25/moving_variance_0',
    'conv_transpose_decoder6/conv1d_transpose_2/kernel_0',
    'conv_transpose_decoder6/conv1d_transpose_2/bias_0',
    'conv_transpose_decoder6/batch_normalization_26/gamma_0',
    'conv_transpose_decoder6/batch_normalization_26/beta_0',
    'conv_transpose_decoder6/batch_normalization_26/moving_mean_0',
    'conv_transpose_decoder6/batch_normalization_26/moving_variance_0',
    'two_conv_decoder6/conv1d_24/kernel_0',
    'two_conv_decoder6/conv1d_24/bias_0',
    'two_conv_decoder6/batch_normalization_27/gamma_0',
    'two_conv_decoder6/batch_normalization_27/beta_0',
    'two_conv_decoder6/batch_normalization_27/moving_mean_0',
    'two_conv_decoder6/batch_normalization_27/moving_variance_0',
    'two_conv_decoder6/conv1d_25/kernel_0',
    'two_conv_decoder6/conv1d_25/bias_0',
    'two_conv_decoder6/batch_normalization_28/gamma_0',
    'two_conv_decoder6/batch_normalization_28/beta_0',
    'two_conv_decoder6/batch_normalization_28/moving_mean_0',
    'two_conv_decoder6/batch_normalization_28/moving_variance_0',
    'conv_transpose_decoder5/conv1d_transpose_3/kernel_0',
    'conv_transpose_decoder5/conv1d_transpose_3/bias_0',
    'conv_transpose_decoder5/batch_normalization_29/gamma_0',
    'conv_transpose_decoder5/batch_normalization_29/beta_0',
    'conv_transpose_decoder5/batch_normalization_29/moving_mean_0',
    'conv_transpose_decoder5/batch_normalization_29/moving_variance_0',
    'two_conv_decoder5/conv1d_26/kernel_0',
    'two_conv_decoder5/conv1d_26/bias_0',
    'two_conv_decoder5/batch_normalization_30/gamma_0',
    'two_conv_decoder5/batch_normalization_30/beta_0',
    'two_conv_decoder5/batch_normalization_30/moving_mean_0',
    'two_conv_decoder5/batch_normalization_30/moving_variance_0',
    'two_conv_decoder5/conv1d_27/kernel_0',
    'two_conv_decoder5/conv1d_27/bias_0',
    'two_conv_decoder5/batch_normalization_31/gamma_0',
    'two_conv_decoder5/batch_normalization_31/beta_0',
    'two_conv_decoder5/batch_normalization_31/moving_mean_0',
    'two_conv_decoder5/batch_normalization_31/moving_variance_0',
    'conv_transpose_decoder4/conv1d_transpose_4/kernel_0',
    'conv_transpose_decoder4/conv1d_transpose_4/bias_0',
    'conv_transpose_decoder4/batch_normalization_32/gamma_0',
    'conv_transpose_decoder4/batch_normalization_32/beta_0',
    'conv_transpose_decoder4/batch_normalization_32/moving_mean_0',
    'conv_transpose_decoder4/batch_normalization_32/moving_variance_0',
    'two_conv_decoder4/conv1d_28/kernel_0',
    'two_conv_decoder4/conv1d_28/bias_0',
    'two_conv_decoder4/batch_normalization_33/gamma_0',
    'two_conv_decoder4/batch_normalization_33/beta_0',
    'two_conv_decoder4/batch_normalization_33/moving_mean_0',
    'two_conv_decoder4/batch_normalization_33/moving_variance_0',
    'two_conv_decoder4/conv1d_29/kernel_0',
    'two_conv_decoder4/conv1d_29/bias_0',
    'two_conv_decoder4/batch_normalization_34/gamma_0',
    'two_conv_decoder4/batch_normalization_34/beta_0',
    'two_conv_decoder4/batch_normalization_34/moving_mean_0',
    'two_conv_decoder4/batch_normalization_34/moving_variance_0',
    'conv_transpose_decoder3/conv1d_transpose_5/kernel_0',
    'conv_transpose_decoder3/conv1d_transpose_5/bias_0',
    'conv_transpose_decoder3/batch_normalization_35/gamma_0',
    'conv_transpose_decoder3/batch_normalization_35/beta_0',
    'conv_transpose_decoder3/batch_normalization_35/moving_mean_0',
    'conv_transpose_decoder3/batch_normalization_35/moving_variance_0',
    'two_conv_decoder3/conv1d_30/kernel_0',
    'two_conv_decoder3/conv1d_30/bias_0',
    'two_conv_decoder3/batch_normalization_36/gamma_0',
    'two_conv_decoder3/batch_normalization_36/beta_0',
    'two_conv_decoder3/batch_normalization_36/moving_mean_0',
    'two_conv_decoder3/batch_normalization_36/moving_variance_0',
    'two_conv_decoder3/conv1d_31/kernel_0',
    'two_conv_decoder3/conv1d_31/bias_0',
    'two_conv_decoder3/batch_normalization_37/gamma_0',
    'two_conv_decoder3/batch_normalization_37/beta_0',
    'two_conv_decoder3/batch_normalization_37/moving_mean_0',
    'two_conv_decoder3/batch_normalization_37/moving_variance_0',
    'conv_transpose_decoder2/conv1d_transpose_6/kernel_0',
    'conv_transpose_decoder2/conv1d_transpose_6/bias_0',
    'conv_transpose_decoder2/batch_normalization_38/gamma_0',
    'conv_transpose_decoder2/batch_normalization_38/beta_0',
    'conv_transpose_decoder2/batch_normalization_38/moving_mean_0',
    'conv_transpose_decoder2/batch_normalization_38/moving_variance_0',
    'two_conv_decoder2/conv1d_32/kernel_0',
    'two_conv_decoder2/conv1d_32/bias_0',
    'two_conv_decoder2/batch_normalization_39/gamma_0',
    'two_conv_decoder2/batch_normalization_39/beta_0',
    'two_conv_decoder2/batch_normalization_39/moving_mean_0',
    'two_conv_decoder2/batch_normalization_39/moving_variance_0',
    'two_conv_decoder2/conv1d_33/kernel_0',
    'two_conv_decoder2/conv1d_33/bias_0',
    'two_conv_decoder2/batch_normalization_40/gamma_0',
    'two_conv_decoder2/batch_normalization_40/beta_0',
    'two_conv_decoder2/batch_normalization_40/moving_mean_0',
    'two_conv_decoder2/batch_normalization_40/moving_variance_0',
    'conv_transpose_decoder1/conv1d_transpose_7/kernel_0',
    'conv_transpose_decoder1/conv1d_transpose_7/bias_0',
    'conv_transpose_decoder1/batch_normalization_41/gamma_0',
    'conv_transpose_decoder1/batch_normalization_41/beta_0',
    'conv_transpose_decoder1/batch_normalization_41/moving_mean_0',
    'conv_transpose_decoder1/batch_normalization_41/moving_variance_0',
    'two_conv_decoder1/conv1d_34/kernel_0',
    'two_conv_decoder1/conv1d_34/bias_0',
    'two_conv_decoder1/batch_normalization_42/gamma_0',
    'two_conv_decoder1/batch_normalization_42/beta_0',
    'two_conv_decoder1/batch_normalization_42/moving_mean_0',
    'two_conv_decoder1/batch_normalization_42/moving_variance_0',
    'two_conv_decoder1/conv1d_35/kernel_0',
    'two_conv_decoder1/conv1d_35/bias_0',
    'two_conv_decoder1/batch_normalization_43/gamma_0',
    'two_conv_decoder1/batch_normalization_43/beta_0',
    'two_conv_decoder1/batch_normalization_43/moving_mean_0',
    'two_conv_decoder1/batch_normalization_43/moving_variance_0',
    'conv_transpose_decoder0/conv1d_transpose_8/kernel_0',
    'conv_transpose_decoder0/conv1d_transpose_8/bias_0',
    'conv_transpose_decoder0/batch_normalization_44/gamma_0',
    'conv_transpose_decoder0/batch_normalization_44/beta_0',
    'conv_transpose_decoder0/batch_normalization_44/moving_mean_0',
    'conv_transpose_decoder0/batch_normalization_44/moving_variance_0',
    'two_conv_decoder0/conv1d_36/kernel_0',
    'two_conv_decoder0/conv1d_36/bias_0',
    'two_conv_decoder0/batch_normalization_45/gamma_0',
    'two_conv_decoder0/batch_normalization_45/beta_0',
    'two_conv_decoder0/batch_normalization_45/moving_mean_0',
    'two_conv_decoder0/batch_normalization_45/moving_variance_0',
    'two_conv_decoder0/conv1d_37/kernel_0',
    'two_conv_decoder0/conv1d_37/bias_0',
    'two_conv_decoder0/batch_normalization_46/gamma_0',
    'two_conv_decoder0/batch_normalization_46/beta_0',
    'two_conv_decoder0/batch_normalization_46/moving_mean_0',
    'two_conv_decoder0/batch_normalization_46/moving_variance_0',
    'conv1d_38/kernel_0',
    'conv1d_38/bias_0'],
   'scalars': ['epoch_loss',
    'epoch_tp',
    'epoch_fp',
    'epoch_tn',
    'epoch_fn',
    'epoch_precision',
    'epoch_recall',
    'epoch_accuracy',
    'epoch_auc'],
   'distributions': ['encode0/conv1d/kernel_0',
    'encode0/conv1d/bias_0',
    'encode0/batch_normalization/gamma_0',
    'encode0/batch_normalization/beta_0',
    'encode0/batch_normalization/moving_mean_0',
    'encode0/batch_normalization/moving_variance_0',
    'encode0/conv1d_1/kernel_0',
    'encode0/conv1d_1/bias_0',
    'encode0/batch_normalization_1/gamma_0',
    'encode0/batch_normalization_1/beta_0',
    'encode0/batch_normalization_1/moving_mean_0',
    'encode0/batch_normalization_1/moving_variance_0',
    'encode1/conv1d_2/kernel_0',
    'encode1/conv1d_2/bias_0',
    'encode1/batch_normalization_2/gamma_0',
    'encode1/batch_normalization_2/beta_0',
    'encode1/batch_normalization_2/moving_mean_0',
    'encode1/batch_normalization_2/moving_variance_0',
    'encode1/conv1d_3/kernel_0',
    'encode1/conv1d_3/bias_0',
    'encode1/batch_normalization_3/gamma_0',
    'encode1/batch_normalization_3/beta_0',
    'encode1/batch_normalization_3/moving_mean_0',
    'encode1/batch_normalization_3/moving_variance_0',
    'encode2/conv1d_4/kernel_0',
    'encode2/conv1d_4/bias_0',
    'encode2/batch_normalization_4/gamma_0',
    'encode2/batch_normalization_4/beta_0',
    'encode2/batch_normalization_4/moving_mean_0',
    'encode2/batch_normalization_4/moving_variance_0',
    'encode2/conv1d_5/kernel_0',
    'encode2/conv1d_5/bias_0',
    'encode2/batch_normalization_5/gamma_0',
    'encode2/batch_normalization_5/beta_0',
    'encode2/batch_normalization_5/moving_mean_0',
    'encode2/batch_normalization_5/moving_variance_0',
    'encode3/conv1d_6/kernel_0',
    'encode3/conv1d_6/bias_0',
    'encode3/batch_normalization_6/gamma_0',
    'encode3/batch_normalization_6/beta_0',
    'encode3/batch_normalization_6/moving_mean_0',
    'encode3/batch_normalization_6/moving_variance_0',
    'encode3/conv1d_7/kernel_0',
    'encode3/conv1d_7/bias_0',
    'encode3/batch_normalization_7/gamma_0',
    'encode3/batch_normalization_7/beta_0',
    'encode3/batch_normalization_7/moving_mean_0',
    'encode3/batch_normalization_7/moving_variance_0',
    'encode4/conv1d_8/kernel_0',
    'encode4/conv1d_8/bias_0',
    'encode4/batch_normalization_8/gamma_0',
    'encode4/batch_normalization_8/beta_0',
    'encode4/batch_normalization_8/moving_mean_0',
    'encode4/batch_normalization_8/moving_variance_0',
    'encode4/conv1d_9/kernel_0',
    'encode4/conv1d_9/bias_0',
    'encode4/batch_normalization_9/gamma_0',
    'encode4/batch_normalization_9/beta_0',
    'encode4/batch_normalization_9/moving_mean_0',
    'encode4/batch_normalization_9/moving_variance_0',
    'encode5/conv1d_10/kernel_0',
    'encode5/conv1d_10/bias_0',
    'encode5/batch_normalization_10/gamma_0',
    'encode5/batch_normalization_10/beta_0',
    'encode5/batch_normalization_10/moving_mean_0',
    'encode5/batch_normalization_10/moving_variance_0',
    'encode5/conv1d_11/kernel_0',
    'encode5/conv1d_11/bias_0',
    'encode5/batch_normalization_11/gamma_0',
    'encode5/batch_normalization_11/beta_0',
    'encode5/batch_normalization_11/moving_mean_0',
    'encode5/batch_normalization_11/moving_variance_0',
    'encode6/conv1d_12/kernel_0',
    'encode6/conv1d_12/bias_0',
    'encode6/batch_normalization_12/gamma_0',
    'encode6/batch_normalization_12/beta_0',
    'encode6/batch_normalization_12/moving_mean_0',
    'encode6/batch_normalization_12/moving_variance_0',
    'encode6/conv1d_13/kernel_0',
    'encode6/conv1d_13/bias_0',
    'encode6/batch_normalization_13/gamma_0',
    'encode6/batch_normalization_13/beta_0',
    'encode6/batch_normalization_13/moving_mean_0',
    'encode6/batch_normalization_13/moving_variance_0',
    'encode7/conv1d_14/kernel_0',
    'encode7/conv1d_14/bias_0',
    'encode7/batch_normalization_14/gamma_0',
    'encode7/batch_normalization_14/beta_0',
    'encode7/batch_normalization_14/moving_mean_0',
    'encode7/batch_normalization_14/moving_variance_0',
    'encode7/conv1d_15/kernel_0',
    'encode7/conv1d_15/bias_0',
    'encode7/batch_normalization_15/gamma_0',
    'encode7/batch_normalization_15/beta_0',
    'encode7/batch_normalization_15/moving_mean_0',
    'encode7/batch_normalization_15/moving_variance_0',
    'encode8/conv1d_16/kernel_0',
    'encode8/conv1d_16/bias_0',
    'encode8/batch_normalization_16/gamma_0',
    'encode8/batch_normalization_16/beta_0',
    'encode8/batch_normalization_16/moving_mean_0',
    'encode8/batch_normalization_16/moving_variance_0',
    'encode8/conv1d_17/kernel_0',
    'encode8/conv1d_17/bias_0',
    'encode8/batch_normalization_17/gamma_0',
    'encode8/batch_normalization_17/beta_0',
    'encode8/batch_normalization_17/moving_mean_0',
    'encode8/batch_normalization_17/moving_variance_0',
    'two_conv_center/conv1d_18/kernel_0',
    'two_conv_center/conv1d_18/bias_0',
    'two_conv_center/batch_normalization_18/gamma_0',
    'two_conv_center/batch_normalization_18/beta_0',
    'two_conv_center/batch_normalization_18/moving_mean_0',
    'two_conv_center/batch_normalization_18/moving_variance_0',
    'two_conv_center/conv1d_19/kernel_0',
    'two_conv_center/conv1d_19/bias_0',
    'two_conv_center/batch_normalization_19/gamma_0',
    'two_conv_center/batch_normalization_19/beta_0',
    'two_conv_center/batch_normalization_19/moving_mean_0',
    'two_conv_center/batch_normalization_19/moving_variance_0',
    'conv_transpose_decoder8/conv1d_transpose/kernel_0',
    'conv_transpose_decoder8/conv1d_transpose/bias_0',
    'conv_transpose_decoder8/batch_normalization_20/gamma_0',
    'conv_transpose_decoder8/batch_normalization_20/beta_0',
    'conv_transpose_decoder8/batch_normalization_20/moving_mean_0',
    'conv_transpose_decoder8/batch_normalization_20/moving_variance_0',
    'two_conv_decoder8/conv1d_20/kernel_0',
    'two_conv_decoder8/conv1d_20/bias_0',
    'two_conv_decoder8/batch_normalization_21/gamma_0',
    'two_conv_decoder8/batch_normalization_21/beta_0',
    'two_conv_decoder8/batch_normalization_21/moving_mean_0',
    'two_conv_decoder8/batch_normalization_21/moving_variance_0',
    'two_conv_decoder8/conv1d_21/kernel_0',
    'two_conv_decoder8/conv1d_21/bias_0',
    'two_conv_decoder8/batch_normalization_22/gamma_0',
    'two_conv_decoder8/batch_normalization_22/beta_0',
    'two_conv_decoder8/batch_normalization_22/moving_mean_0',
    'two_conv_decoder8/batch_normalization_22/moving_variance_0',
    'conv_transpose_decoder7/conv1d_transpose_1/kernel_0',
    'conv_transpose_decoder7/conv1d_transpose_1/bias_0',
    'conv_transpose_decoder7/batch_normalization_23/gamma_0',
    'conv_transpose_decoder7/batch_normalization_23/beta_0',
    'conv_transpose_decoder7/batch_normalization_23/moving_mean_0',
    'conv_transpose_decoder7/batch_normalization_23/moving_variance_0',
    'two_conv_decoder7/conv1d_22/kernel_0',
    'two_conv_decoder7/conv1d_22/bias_0',
    'two_conv_decoder7/batch_normalization_24/gamma_0',
    'two_conv_decoder7/batch_normalization_24/beta_0',
    'two_conv_decoder7/batch_normalization_24/moving_mean_0',
    'two_conv_decoder7/batch_normalization_24/moving_variance_0',
    'two_conv_decoder7/conv1d_23/kernel_0',
    'two_conv_decoder7/conv1d_23/bias_0',
    'two_conv_decoder7/batch_normalization_25/gamma_0',
    'two_conv_decoder7/batch_normalization_25/beta_0',
    'two_conv_decoder7/batch_normalization_25/moving_mean_0',
    'two_conv_decoder7/batch_normalization_25/moving_variance_0',
    'conv_transpose_decoder6/conv1d_transpose_2/kernel_0',
    'conv_transpose_decoder6/conv1d_transpose_2/bias_0',
    'conv_transpose_decoder6/batch_normalization_26/gamma_0',
    'conv_transpose_decoder6/batch_normalization_26/beta_0',
    'conv_transpose_decoder6/batch_normalization_26/moving_mean_0',
    'conv_transpose_decoder6/batch_normalization_26/moving_variance_0',
    'two_conv_decoder6/conv1d_24/kernel_0',
    'two_conv_decoder6/conv1d_24/bias_0',
    'two_conv_decoder6/batch_normalization_27/gamma_0',
    'two_conv_decoder6/batch_normalization_27/beta_0',
    'two_conv_decoder6/batch_normalization_27/moving_mean_0',
    'two_conv_decoder6/batch_normalization_27/moving_variance_0',
    'two_conv_decoder6/conv1d_25/kernel_0',
    'two_conv_decoder6/conv1d_25/bias_0',
    'two_conv_decoder6/batch_normalization_28/gamma_0',
    'two_conv_decoder6/batch_normalization_28/beta_0',
    'two_conv_decoder6/batch_normalization_28/moving_mean_0',
    'two_conv_decoder6/batch_normalization_28/moving_variance_0',
    'conv_transpose_decoder5/conv1d_transpose_3/kernel_0',
    'conv_transpose_decoder5/conv1d_transpose_3/bias_0',
    'conv_transpose_decoder5/batch_normalization_29/gamma_0',
    'conv_transpose_decoder5/batch_normalization_29/beta_0',
    'conv_transpose_decoder5/batch_normalization_29/moving_mean_0',
    'conv_transpose_decoder5/batch_normalization_29/moving_variance_0',
    'two_conv_decoder5/conv1d_26/kernel_0',
    'two_conv_decoder5/conv1d_26/bias_0',
    'two_conv_decoder5/batch_normalization_30/gamma_0',
    'two_conv_decoder5/batch_normalization_30/beta_0',
    'two_conv_decoder5/batch_normalization_30/moving_mean_0',
    'two_conv_decoder5/batch_normalization_30/moving_variance_0',
    'two_conv_decoder5/conv1d_27/kernel_0',
    'two_conv_decoder5/conv1d_27/bias_0',
    'two_conv_decoder5/batch_normalization_31/gamma_0',
    'two_conv_decoder5/batch_normalization_31/beta_0',
    'two_conv_decoder5/batch_normalization_31/moving_mean_0',
    'two_conv_decoder5/batch_normalization_31/moving_variance_0',
    'conv_transpose_decoder4/conv1d_transpose_4/kernel_0',
    'conv_transpose_decoder4/conv1d_transpose_4/bias_0',
    'conv_transpose_decoder4/batch_normalization_32/gamma_0',
    'conv_transpose_decoder4/batch_normalization_32/beta_0',
    'conv_transpose_decoder4/batch_normalization_32/moving_mean_0',
    'conv_transpose_decoder4/batch_normalization_32/moving_variance_0',
    'two_conv_decoder4/conv1d_28/kernel_0',
    'two_conv_decoder4/conv1d_28/bias_0',
    'two_conv_decoder4/batch_normalization_33/gamma_0',
    'two_conv_decoder4/batch_normalization_33/beta_0',
    'two_conv_decoder4/batch_normalization_33/moving_mean_0',
    'two_conv_decoder4/batch_normalization_33/moving_variance_0',
    'two_conv_decoder4/conv1d_29/kernel_0',
    'two_conv_decoder4/conv1d_29/bias_0',
    'two_conv_decoder4/batch_normalization_34/gamma_0',
    'two_conv_decoder4/batch_normalization_34/beta_0',
    'two_conv_decoder4/batch_normalization_34/moving_mean_0',
    'two_conv_decoder4/batch_normalization_34/moving_variance_0',
    'conv_transpose_decoder3/conv1d_transpose_5/kernel_0',
    'conv_transpose_decoder3/conv1d_transpose_5/bias_0',
    'conv_transpose_decoder3/batch_normalization_35/gamma_0',
    'conv_transpose_decoder3/batch_normalization_35/beta_0',
    'conv_transpose_decoder3/batch_normalization_35/moving_mean_0',
    'conv_transpose_decoder3/batch_normalization_35/moving_variance_0',
    'two_conv_decoder3/conv1d_30/kernel_0',
    'two_conv_decoder3/conv1d_30/bias_0',
    'two_conv_decoder3/batch_normalization_36/gamma_0',
    'two_conv_decoder3/batch_normalization_36/beta_0',
    'two_conv_decoder3/batch_normalization_36/moving_mean_0',
    'two_conv_decoder3/batch_normalization_36/moving_variance_0',
    'two_conv_decoder3/conv1d_31/kernel_0',
    'two_conv_decoder3/conv1d_31/bias_0',
    'two_conv_decoder3/batch_normalization_37/gamma_0',
    'two_conv_decoder3/batch_normalization_37/beta_0',
    'two_conv_decoder3/batch_normalization_37/moving_mean_0',
    'two_conv_decoder3/batch_normalization_37/moving_variance_0',
    'conv_transpose_decoder2/conv1d_transpose_6/kernel_0',
    'conv_transpose_decoder2/conv1d_transpose_6/bias_0',
    'conv_transpose_decoder2/batch_normalization_38/gamma_0',
    'conv_transpose_decoder2/batch_normalization_38/beta_0',
    'conv_transpose_decoder2/batch_normalization_38/moving_mean_0',
    'conv_transpose_decoder2/batch_normalization_38/moving_variance_0',
    'two_conv_decoder2/conv1d_32/kernel_0',
    'two_conv_decoder2/conv1d_32/bias_0',
    'two_conv_decoder2/batch_normalization_39/gamma_0',
    'two_conv_decoder2/batch_normalization_39/beta_0',
    'two_conv_decoder2/batch_normalization_39/moving_mean_0',
    'two_conv_decoder2/batch_normalization_39/moving_variance_0',
    'two_conv_decoder2/conv1d_33/kernel_0',
    'two_conv_decoder2/conv1d_33/bias_0',
    'two_conv_decoder2/batch_normalization_40/gamma_0',
    'two_conv_decoder2/batch_normalization_40/beta_0',
    'two_conv_decoder2/batch_normalization_40/moving_mean_0',
    'two_conv_decoder2/batch_normalization_40/moving_variance_0',
    'conv_transpose_decoder1/conv1d_transpose_7/kernel_0',
    'conv_transpose_decoder1/conv1d_transpose_7/bias_0',
    'conv_transpose_decoder1/batch_normalization_41/gamma_0',
    'conv_transpose_decoder1/batch_normalization_41/beta_0',
    'conv_transpose_decoder1/batch_normalization_41/moving_mean_0',
    'conv_transpose_decoder1/batch_normalization_41/moving_variance_0',
    'two_conv_decoder1/conv1d_34/kernel_0',
    'two_conv_decoder1/conv1d_34/bias_0',
    'two_conv_decoder1/batch_normalization_42/gamma_0',
    'two_conv_decoder1/batch_normalization_42/beta_0',
    'two_conv_decoder1/batch_normalization_42/moving_mean_0',
    'two_conv_decoder1/batch_normalization_42/moving_variance_0',
    'two_conv_decoder1/conv1d_35/kernel_0',
    'two_conv_decoder1/conv1d_35/bias_0',
    'two_conv_decoder1/batch_normalization_43/gamma_0',
    'two_conv_decoder1/batch_normalization_43/beta_0',
    'two_conv_decoder1/batch_normalization_43/moving_mean_0',
    'two_conv_decoder1/batch_normalization_43/moving_variance_0',
    'conv_transpose_decoder0/conv1d_transpose_8/kernel_0',
    'conv_transpose_decoder0/conv1d_transpose_8/bias_0',
    'conv_transpose_decoder0/batch_normalization_44/gamma_0',
    'conv_transpose_decoder0/batch_normalization_44/beta_0',
    'conv_transpose_decoder0/batch_normalization_44/moving_mean_0',
    'conv_transpose_decoder0/batch_normalization_44/moving_variance_0',
    'two_conv_decoder0/conv1d_36/kernel_0',
    'two_conv_decoder0/conv1d_36/bias_0',
    'two_conv_decoder0/batch_normalization_45/gamma_0',
    'two_conv_decoder0/batch_normalization_45/beta_0',
    'two_conv_decoder0/batch_normalization_45/moving_mean_0',
    'two_conv_decoder0/batch_normalization_45/moving_variance_0',
    'two_conv_decoder0/conv1d_37/kernel_0',
    'two_conv_decoder0/conv1d_37/bias_0',
    'two_conv_decoder0/batch_normalization_46/gamma_0',
    'two_conv_decoder0/batch_normalization_46/beta_0',
    'two_conv_decoder0/batch_normalization_46/moving_mean_0',
    'two_conv_decoder0/batch_normalization_46/moving_variance_0',
    'conv1d_38/kernel_0',
    'conv1d_38/bias_0'],
   'tensors': ['batch_2'],
   'graph': False,
   'meta_graph': False,
   'run_metadata': []}
ea.Scalars('epoch_loss')

2.1.8 Load model from mlflow logs and predict separately loaded data

First, check if the model file has to be downloaded via git lfs

cd /home/lex/Programme/drmed-git/
git lfs ls-files
git lfs checkout
(base) [lex@Topialex drmed-git]$ git lfs ls-files
6526c5abca - data/mlruns/1/47b870b8fdcb4445956635c6758caff3/artifacts/model/data/model.h5

(base) [lex@Topialex drmed-git]$ git lfs checkout
Skipped checkout for
"data/mlruns/1/47b870b8fdcb4445956635c6758caff3/artifacts/model/data/model.h5",
content not local. Use fetch to download.
Checking out LFS objects: 100% (1/1), 399 MB | 0 B/s, done.
git lfs pull
fetch: Fetching reference refs/heads/develop
Username for 'https://github.com': aseltmann B/s
Password for 'https://aseltmann@github.com':
(base) [lex@Topialex drmed-git]$ 1), 399 MB | 1.9 MB/s

Note: after these downloads, the GitHub-Version of git lfs didn't work for me anymore without a payment for download/upload. Initially I thought of git lfs as a distributed way of storing files without the need of a central storage. GitHub does not seem to support that, which is sad - I will move away from it in the future. See here.

import mlflow.keras
import mlflow.tensorflow
import sys
import numpy as np
import tensorflow as tf

project_path = '/home/lex/Programme/drmed-git'
fluotracify_path = '{}/src/'.format(project_path)
sys.path.append(fluotracify_path)

from fluotracify.simulations import import_simulation_from_csv as isfc
from fluotracify.training import build_model as bm, preprocess_data as ppd

%cd /home/lex/Programme/drmed-git
print(tf.__version__)
/home/lex/Programme/drmed-git
2.3.0-dev20200519
mlflow.set_tracking_uri('file://{}/data/mlruns'.format(project_path))
client = mlflow.tracking.MlflowClient(tracking_uri=mlflow.get_tracking_uri())
model_path = client.download_artifacts(
    run_id='1aefda1366f04f5da5d1fc2241ad9208',
    path='model')
print(model_path)
/home/lex/Programme/drmed-git/data/mlruns/0/1aefda1366f04f5da5d1fc2241ad9208/artifacts/model
bm.binary_ce_dice_loss()
<function fluotracify.training.build_model.binary_ce_dice_loss.<locals>.binary_ce_dice(y_true, y_pred)>
# mlflow.keras module
model_keras = mlflow.keras.load_model(model_uri=model_path,
                                      custom_objects={'binary_ce_dice': bm.binary_ce_dice_loss()})
model_keras
<tensorflow.python.keras.engine.functional.Functional at 0x7f3afa29d0a0>
data, _, nsamples, experiment_params = isfc.import_from_csv(
    path='/home/lex/Programme/Jupyter/DOKTOR/saves/firstartefact/subsample_rand/',
    header=12,
    frac_train=1,
    col_per_example=2,
    dropindex=None,
    dropcolumns='Unnamed: 200')
train 0 /home/lex/Programme/Jupyter/DOKTOR/saves/firstartefact/subsample_rand/traces_brightclust_rand_Sep2019_set003.csv
train 1 /home/lex/Programme/Jupyter/DOKTOR/saves/firstartefact/subsample_rand/traces_brightclust_rand_Sep2019_set002.csv
train 2 /home/lex/Programme/Jupyter/DOKTOR/saves/firstartefact/subsample_rand/traces_brightclust_rand_Sep2019_set001.csv
input - shape:	 (None, 16384, 1)
output - shape:	 (None, 16384, 1)
Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_1 (InputLayer)            [(None, 16384, 1)]   0
__________________________________________________________________________________________________
encode0 (Sequential)            (None, 16384, 64)    13120       input_1[0][0]
__________________________________________________________________________________________________
mp_encode0 (MaxPooling1D)       (None, 8192, 64)     0           encode0[0][0]
__________________________________________________________________________________________________
encode1 (Sequential)            (None, 8192, 128)    75008       mp_encode0[0][0]
__________________________________________________________________________________________________
mp_encode1 (MaxPooling1D)       (None, 4096, 128)    0           encode1[0][0]
__________________________________________________________________________________________________
encode2 (Sequential)            (None, 4096, 256)    297472      mp_encode1[0][0]
__________________________________________________________________________________________________
mp_encode2 (MaxPooling1D)       (None, 2048, 256)    0           encode2[0][0]
__________________________________________________________________________________________________
encode3 (Sequential)            (None, 2048, 512)    1184768     mp_encode2[0][0]
__________________________________________________________________________________________________
mp_encode3 (MaxPooling1D)       (None, 1024, 512)    0           encode3[0][0]
__________________________________________________________________________________________________
encode4 (Sequential)            (None, 1024, 512)    1577984     mp_encode3[0][0]
__________________________________________________________________________________________________
mp_encode4 (MaxPooling1D)       (None, 512, 512)     0           encode4[0][0]
__________________________________________________________________________________________________
encode5 (Sequential)            (None, 512, 512)     1577984     mp_encode4[0][0]
__________________________________________________________________________________________________
mp_encode5 (MaxPooling1D)       (None, 256, 512)     0           encode5[0][0]
__________________________________________________________________________________________________
encode6 (Sequential)            (None, 256, 512)     1577984     mp_encode5[0][0]
__________________________________________________________________________________________________
mp_encode6 (MaxPooling1D)       (None, 128, 512)     0           encode6[0][0]
__________________________________________________________________________________________________
encode7 (Sequential)            (None, 128, 512)     1577984     mp_encode6[0][0]
__________________________________________________________________________________________________
mp_encode7 (MaxPooling1D)       (None, 64, 512)      0           encode7[0][0]
__________________________________________________________________________________________________
encode8 (Sequential)            (None, 64, 512)      1577984     mp_encode7[0][0]
__________________________________________________________________________________________________
mp_encode8 (MaxPooling1D)       (None, 32, 512)      0           encode8[0][0]
__________________________________________________________________________________________________
two_conv_center (Sequential)    (None, 32, 1024)     4728832     mp_encode8[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder8 (Sequen (None, 64, 512)      1051136     two_conv_center[0][0]
__________________________________________________________________________________________________
decoder8 (Concatenate)          (None, 64, 1024)     0           encode8[0][0]
                                                                 conv_transpose_decoder8[0][0]
__________________________________________________________________________________________________
two_conv_decoder8 (Sequential)  (None, 64, 512)      2364416     decoder8[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder7 (Sequen (None, 128, 512)     526848      two_conv_decoder8[0][0]
__________________________________________________________________________________________________
decoder7 (Concatenate)          (None, 128, 1024)    0           encode7[0][0]
                                                                 conv_transpose_decoder7[0][0]
__________________________________________________________________________________________________
two_conv_decoder7 (Sequential)  (None, 128, 512)     2364416     decoder7[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder6 (Sequen (None, 256, 512)     526848      two_conv_decoder7[0][0]
__________________________________________________________________________________________________
decoder6 (Concatenate)          (None, 256, 1024)    0           encode6[0][0]
                                                                 conv_transpose_decoder6[0][0]
__________________________________________________________________________________________________
two_conv_decoder6 (Sequential)  (None, 256, 512)     2364416     decoder6[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder5 (Sequen (None, 512, 512)     526848      two_conv_decoder6[0][0]
__________________________________________________________________________________________________
decoder5 (Concatenate)          (None, 512, 1024)    0           encode5[0][0]
                                                                 conv_transpose_decoder5[0][0]
__________________________________________________________________________________________________
two_conv_decoder5 (Sequential)  (None, 512, 512)     2364416     decoder5[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder4 (Sequen (None, 1024, 512)    526848      two_conv_decoder5[0][0]
__________________________________________________________________________________________________
decoder4 (Concatenate)          (None, 1024, 1024)   0           encode4[0][0]
                                                                 conv_transpose_decoder4[0][0]
__________________________________________________________________________________________________
two_conv_decoder4 (Sequential)  (None, 1024, 512)    2364416     decoder4[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder3 (Sequen (None, 2048, 512)    526848      two_conv_decoder4[0][0]
__________________________________________________________________________________________________
decoder3 (Concatenate)          (None, 2048, 1024)   0           encode3[0][0]
                                                                 conv_transpose_decoder3[0][0]
__________________________________________________________________________________________________
two_conv_decoder3 (Sequential)  (None, 2048, 512)    2364416     decoder3[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder2 (Sequen (None, 4096, 256)    263424      two_conv_decoder3[0][0]
__________________________________________________________________________________________________
decoder2 (Concatenate)          (None, 4096, 512)    0           encode2[0][0]
                                                                 conv_transpose_decoder2[0][0]
__________________________________________________________________________________________________
two_conv_decoder2 (Sequential)  (None, 4096, 256)    592384      decoder2[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder1 (Sequen (None, 8192, 128)    66176       two_conv_decoder2[0][0]
__________________________________________________________________________________________________
decoder1 (Concatenate)          (None, 8192, 256)    0           encode1[0][0]
                                                                 conv_transpose_decoder1[0][0]
__________________________________________________________________________________________________
two_conv_decoder1 (Sequential)  (None, 8192, 128)    148736      decoder1[0][0]
__________________________________________________________________________________________________
conv_transpose_decoder0 (Sequen (None, 16384, 64)    16704       two_conv_decoder1[0][0]
__________________________________________________________________________________________________
decoder0 (Concatenate)          (None, 16384, 128)   0           encode0[0][0]
                                                                 conv_transpose_decoder0[0][0]
__________________________________________________________________________________________________
two_conv_decoder0 (Sequential)  (None, 16384, 64)    37504       decoder0[0][0]
__________________________________________________________________________________________________
conv1d_38 (Conv1D)              (None, 16384, 1)     65          two_conv_decoder0[0][0]
==================================================================================================
Total params: 33,185,985
Trainable params: 33,146,689
Non-trainable params: 39,296
__________________________________________________________________________________________________
None
prediction = model_keras.predict(np.array(data.iloc[:16384, 0]).reshape(1, -1, 1))

I noticed one problem: directly outputting the prediction array for large predictions (e.g. 16384 time steps), will freeze emacs. Especially with my setup with conversion to org tables. It is possible to output it in the REPL, or print it using print(prediction), because there the output gets truncated, e.g. like this:

array([[[1.],
        [1.],
        [1.],
        ...,
        [1.],
        [1.],
        [1.]]], dtype=float32)

While here it tries to print everything (I guess). Maybe I should test the :pandoc t argument instead of the OrgFormatter class

Basically, no matter if I choosnpe the :results header argument, it always gets printed with :display org like this.

log_dir="logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(
    log_dir=log_dir,
    histogram_freq=5,
    write_images=True,
    update_freq='batch')

epochs = 50

history = model.fit(x=dataset_train,
                    epochs=epochs,
                    steps_per_epoch=400,
                    validation_data=dataset_val,
                    validation_steps=tf.math.ceil(num_val_examples / batch_size),
                    callbacks=[tensorboard_callback])
WARNING: Logging before flag parsing goes to stderr.
W0413 03:23:05.926374 47620222508736 summary_ops_v2.py:1132] Model failed to serialize as JSON. Ignoring... Layers with arguments in `__init__` must override `get_config`.
Train for 400 steps, validate for 534.0 steps
Epoch 1/50
  8/400 [..............................] - ETA: 1:05:18 - loss: 1.4586 - mean_io_u: 0.4039 - precision: 0.2347 - recall: 0.1408
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
prediction
  0 1 2 3 4 5 6 7 8 9 10 11 2043 2044 2045 2046 2047
0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

To make the output work the desired way (like in the REPL or print() and without freezing emacs for large tables), use :display plain

prediction
array([[[1.],
        [1.],
        [1.],
        ...,
        [1.],
        [1.],
        [1.]]], dtype=float32)

2.1.9 git exp 2

git status
git log -1
(tensorflow_nightly) [ye53nis@node151 drmed-git]$ git status
# On branch exp-310520-unet
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#  ...
nothing added to commit but untracked files present (use "git add" to track)

(tensorflow_nightly) [ye53nis@node151 drmed-git]$ git log -1
commit 09c1d2a1bc083695026b46c74ea175c9168bb5f2
Author: Apoplex <oligolex@vivaldi.net>
Date:   Fri Jun 12 14:17:14 2020 +0200

    New learning rate schedule

current learning rate schedule:

learning_rate = 0.1
 if epoch > 10:
     learning_rate = 0.01
 if epoch > 20:
     learning_rate = 0.001
 if epoch > 30:
     learning_rate = 0.0001
 if epoch > 40:
     learning_rate = 0.00001
 if epoch > 50:
     learning_rate = 0.000001
 if epoch > 60:
     learning_rate = 0.00000001
 if epoch > 70:
     learning_rate = 0.0000000001
 if epoch > 80:
     learning_rate = 0.000000000001
 if epoch > 90:
     learning_rate = 0.00000000000001

2.1.10 experimental run 2 - full dataset

This time new learning rate

mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=100 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact_Sep2019 -P steps_per_epoch=1280 -P validation_steps=320
(tensorflow_nightly) [ye53nis@node151 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=100 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact_Sep2019 -P steps_per_epoch=1280 -P val
idation_steps=320
2020/06/12 14:20:34 INFO mlflow.projects: === Created directory /tmp/tmp2s79px_z for downloading remote URIs passed to arguments of type 'path' ===
2020/06/12 14:20:34 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/fluotracify/training/train.py /
beegfs/ye53nis/drmed-git/src 5 0.2 16384 None 100 /beegfs/ye53nis/saves/firstartefact_Sep2019 1280 320' in run with ID '037e1d9e4ad74784974f4aaac11138cc' ===
2020-06-12 14:20:53.286502: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-06-12 14:20:53.286593: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favour of importlib; see the modul
e's documentation for alternative uses
  import imp
2.3.0-dev20200527
2020-06-12 14:21:33.782337: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-06-12 14:21:33.782422: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-06-12 14:21:33.782480: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node151): /proc/driver/nvidia/version does not exist
GPUs:  []
train 0 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set027.csv
train 1 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set087.csv
train 2 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set003.csv
train 3 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set056.csv
train 4 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set076.csv
train 5 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set094.csv
train 6 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set017.csv
train 7 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set074.csv
train 8 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set055.csv
train 9 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set096.csv
train 10 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set054.csv
train 11 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set093.csv
train 12 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set079.csv
train 13 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set014.csv
train 14 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set008.csv
train 15 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set031.csv
train 16 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set023.csv
train 17 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set025.csv
train 18 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set034.csv
train 19 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set009.csv
train 20 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set044.csv
train 21 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set063.csv
train 22 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set004.csv
train 23 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set072.csv
train 24 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set046.csv
train 25 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set049.csv
train 26 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set007.csv
train 27 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set100.csv
train 28 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set083.csv
train 29 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set077.csv
train 30 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set061.csv
train 31 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set081.csv
train 32 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set091.csv
train 33 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set069.csv
train 34 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set052.csv
train 35 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set028.csv
train 36 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set019.csv
train 37 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set057.csv
train 38 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set064.csv
train 39 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set075.csv
train 40 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set002.csv
train 41 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set062.csv
train 42 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set043.csv
train 43 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set042.csv
train 44 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set005.csv
train 45 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set016.csv
train 46 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set018.csv
train 47 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set041.csv
train 48 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set039.csv
train 49 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set006.csv
train 50 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set092.csv
train 51 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set060.csv
train 52 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set001.csv
train 53 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set035.csv
train 54 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set029.csv
train 55 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set051.csv
train 56 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set012.csv
train 57 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set036.csv
train 58 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set024.csv
train 59 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set053.csv
train 60 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set011.csv
train 61 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set032.csv
train 62 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set067.csv
train 63 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set058.csv
train 64 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set080.csv
train 65 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set086.csv
train 66 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set033.csv
train 67 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set085.csv
train 68 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set015.csv
train 69 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set090.csv
train 70 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set020.csv
train 71 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set030.csv
train 72 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set050.csv
train 73 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set098.csv
train 74 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set099.csv
train 75 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set070.csv
train 76 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set021.csv
train 77 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set095.csv
train 78 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set073.csv
train 79 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set078.csv
test 80 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set026.csv
test 81 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set038.csv
test 82 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set082.csv
test 83 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set047.csv
test 84 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set040.csv
test 85 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set066.csv
test 86 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set059.csv
test 87 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set013.csv
test 88 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set089.csv
test 89 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set071.csv
test 90 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set088.csv
test 91 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set037.csv
test 92 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set022.csv
test 93 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set084.csv
test 94 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set010.csv
test 95 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set097.csv
test 96 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set068.csv
test 97 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set065.csv
test 98 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set048.csv
test 99 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set045.csv
shapes of feature dataframe: (20000, 8000) and label dataframe: (20000, 8000)
shapes of feature dataframe: (20000, 2000) and label dataframe: (20000, 2000)

for each 20,000 timestap trace there are the following numbers of corrupted timesteps:
 label001_1    6286
label001_1    2568
label001_1    4495
label001_1    4414
label001_1    1105
dtype: int64
2020-06-12 14:26:49.734444: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-06-12 14:26:49.744132: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz
2020-06-12 14:26:49.745558: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55721c34af40 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-12 14:26:49.745586: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
number of training examples: 6400, number of validation examples: 1600

------------------------
number of test examples: 2000

input - shape:   (None, 16384, 1)
output - shape:  (None, 16384, 1)
2020-06-12 14:26:53.653730: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the ABCs from 'collections' instead
 of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  if not isinstance(wrapped_dict, collections.Mapping):
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature()
 or inspect.getfullargspec()
  all_param_names, _, _, all_default_values = inspect.getargspec(fn)  # pylint: disable=W1505
Epoch 1/100
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from
'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  if not isinstance(values, collections.Sequence):
   1/1280 [..............................] - ETA: 0s - loss: 1.5430 - tp: 5048.0000 - fp: 18093.0000 - tn: 50755.0000 - fn: 8024.0000 - precision: 0.2181 - recall: 0.3862 - accuracy: 0.6812 - auc: 0.58482020-06-12 14:27:06.190184: I ten
sorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager.profiler) is deprecated and w
ill be removed after 2020-07-01.
Instructions for updating:
use `tf.profiler.experimental.stop` instead.
2020-06-12 14:27:07.670394: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_06_12_14_27_07
2020-06-12 14:27:07.684980: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_06_12_14_27_07/node151.trace.json.gz
2020-06-12 14:27:07.711539: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_06_12_14_27_07
2020-06-12 14:27:07.711638: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_06_12_14_27_07/node151.memory_profile.json.gz
2020-06-12 14:27:07.713732: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_06_12_14_27_07Dumped tool data for xplane.pb to /tmp/tb/train/plugins/profile/2020_06_12_1
4_27_07/node151.xplane.pb
Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_06_12_14_27_07/node151.overview_page.pb
Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_06_12_14_27_07/node151.input_pipeline.pb
Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_06_12_14_27_07/node151.tensorflow_stats.pb
Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_06_12_14_27_07/node151.kernel_stats.pb

1280/1280 [==============================] - 2053s 2s/step - loss: 0.9576 - tp: 14931613.0000 - fp: 6406557.0000 - tn: 74369864.0000 - fn: 9149518.0000 - precision: 0.6998 - recall: 0.6201 - accuracy: 0.8516 - auc: 0.8805 - val_loss: 1.
1981 - val_tp: 3451021.0000 - val_fp: 465330.0000 - val_tn: 19870236.0000 - val_fn: 2427812.0000 - val_precision: 0.8812 - val_recall: 0.5870 - val_accuracy: 0.8896 - val_auc: 0.8828
Epoch 2/100
 679/1280 [==============>...............] - ETA: 14:47 - loss: 0.6900 - tp: 9867994.0000 - fp: 2756931.0000 - tn: 40089676.0000 - fn: 2909066.0000 - precision: 0.7816 - recall: 0.7723 - accuracy: 0.8981 - auc: 0.9307
1280/1280 [==============================] - 1998s 2s/step - loss: 0.6748 - tp: 18707488.0000 - fp: 4980630.0000 - tn: 75750952.0000 - fn: 5418544.0000 - precision: 0.7897 - recall: 0.7754 - accuracy: 0.9008 - auc: 0.9335 - val_loss: 1.
8923 - val_tp: 1511411.0000 - val_fp: 71511.0000 - val_tn: 20387136.0000 - val_fn: 4244342.0000 - val_precision: 0.9548 - val_recall: 0.2626 - val_accuracy: 0.8354 - val_auc: 0.8004
Epoch 3/100
1280/1280 [==============================] - 1974s 2s/step - loss: 0.6086 - tp: 19388796.0000 - fp: 4672424.0000 - tn: 76028144.0000 - fn: 4768219.0000 - precision: 0.8058 - recall: 0.8026 - accuracy: 0.9100 - auc: 0.9440 - val_loss: 4.
4772 - val_tp: 497608.0000 - val_fp: 47.0000 - val_tn: 20133974.0000 - val_fn: 5582775.0000 - val_precision: 0.9999 - val_recall: 0.0818 - val_accuracy: 0.7870 - val_auc: 0.6095
Epoch 4/100
1280/1280 [==============================] - 2036s 2s/step - loss: 0.5105 - tp: 20225824.0000 - fp: 3816004.0000 - tn: 76833904.0000 - fn: 3981872.0000 - precision: 0.8413 - recall: 0.8355 - accuracy: 0.9256 - auc: 0.9568 - val_loss: 1.
5316 - val_tp: 1395128.0000 - val_fp: 5102.0000 - val_tn: 20164212.0000 - val_fn: 4649960.0000 - val_precision: 0.9964 - val_recall: 0.2308 - val_accuracy: 0.8224 - val_auc: 0.7861
Epoch 5/100
1280/1280 [==============================] - 1961s 2s/step - loss: 0.4310 - tp: 20871492.0000 - fp: 3257518.0000 - tn: 77358976.0000 - fn: 3369640.0000 - precision: 0.8650 - recall: 0.8610 - accuracy: 0.9368 - auc: 0.9655 - val_loss: 0.
6155 - val_tp: 4878880.0000 - val_fp: 1771345.0000 - val_tn: 18827236.0000 - val_fn: 736940.0000 - val_precision: 0.7336 - val_recall: 0.8688 - val_accuracy: 0.9043 - val_auc: 0.9524
Epoch 6/100
1280/1280 [==============================] - 1934s 2s/step - loss: 0.3911 - tp: 20849656.0000 - fp: 2983615.0000 - tn: 77906056.0000 - fn: 3118252.0000 - precision: 0.8748 - recall: 0.8699 - accuracy: 0.9418 - auc: 0.9695 - val_loss: 13
.7751 - val_tp: 5829853.0000 - val_fp: 12729657.0000 - val_tn: 7490981.0000 - val_fn: 163909.0000 - val_precision: 0.3141 - val_recall: 0.9727 - val_accuracy: 0.5081 - val_auc: 0.7568
Epoch 7/100
1280/1280 [==============================] - 1954s 2s/step - loss: 0.3763 - tp: 20945158.0000 - fp: 2869675.0000 - tn: 78062216.0000 - fn: 2980532.0000 - precision: 0.8795 - recall: 0.8754 - accuracy: 0.9442 - auc: 0.9712 - val_loss: 1.
4221 - val_tp: 2763303.0000 - val_fp: 537658.0000 - val_tn: 19870444.0000 - val_fn: 3042995.0000 - val_precision: 0.8371 - val_recall: 0.4759 - val_accuracy: 0.8634 - val_auc: 0.8209
Epoch 8/100
1280/1280 [==============================] - 1959s 2s/step - loss: 0.3565 - tp: 21224852.0000 - fp: 2648995.0000 - tn: 78077912.0000 - fn: 2905816.0000 - precision: 0.8890 - recall: 0.8796 - accuracy: 0.9470 - auc: 0.9734 - val_loss: 4.
1428 - val_tp: 671439.0000 - val_fp: 5.0000 - val_tn: 19926294.0000 - val_fn: 5616654.0000 - val_precision: 1.0000 - val_recall: 0.1068 - val_accuracy: 0.7857 - val_auc: 0.6147
Epoch 9/100
1280/1280 [==============================] - 1959s 2s/step - loss: 0.3532 - tp: 21099540.0000 - fp: 2612022.0000 - tn: 78277824.0000 - fn: 2868197.0000 - precision: 0.8898 - recall: 0.8803 - accuracy: 0.9477 - auc: 0.9736 - val_loss: 0.
4181 - val_tp: 5276605.0000 - val_fp: 1364866.0000 - val_tn: 19070268.0000 - val_fn: 502655.0000 - val_precision: 0.7945 - val_recall: 0.9130 - val_accuracy: 0.9288 - val_auc: 0.9733
Epoch 10/100
1280/1280 [==============================] - 1964s 2s/step - loss: 0.3303 - tp: 21854000.0000 - fp: 2601074.0000 - tn: 77778032.0000 - fn: 2624482.0000 - precision: 0.8936 - recall: 0.8928 - accuracy: 0.9502 - auc: 0.9758 - val_loss: 0.
8186 - val_tp: 4852372.0000 - val_fp: 1259191.0000 - val_tn: 19046032.0000 - val_fn: 1056801.0000 - val_precision: 0.7940 - val_recall: 0.8212 - val_accuracy: 0.9117 - val_auc: 0.9279
Epoch 11/100
1280/1280 [==============================] - 1954s 2s/step - loss: 0.3367 - tp: 21247744.0000 - fp: 2633337.0000 - tn: 78318176.0000 - fn: 2658349.0000 - precision: 0.8897 - recall: 0.8888 - accuracy: 0.9495 - auc: 0.9748 - val_loss: 14
.1746 - val_tp: 5315183.0000 - val_fp: 9292374.0000 - val_tn: 10878559.0000 - val_fn: 728284.0000 - val_precision: 0.3639 - val_recall: 0.8795 - val_accuracy: 0.6177 - val_auc: 0.7350
Epoch 12/100
1280/1280 [==============================] - 1962s 2s/step - loss: 0.3097 - tp: 21383088.0000 - fp: 2165215.0000 - tn: 78747360.0000 - fn: 2561919.0000 - precision: 0.9081 - recall: 0.8930 - accuracy: 0.9549 - auc: 0.9777 - val_loss: 0.
3274 - val_tp: 4925695.0000 - val_fp: 165902.0000 - val_tn: 20004090.0000 - val_fn: 1118702.0000 - val_precision: 0.9674 - val_recall: 0.8149 - val_accuracy: 0.9510 - val_auc: 0.9759
Epoch 13/100
1280/1280 [==============================] - 1969s 2s/step - loss: 0.2912 - tp: 21857506.0000 - fp: 2213488.0000 - tn: 78446488.0000 - fn: 2340152.0000 - precision: 0.9080 - recall: 0.9033 - accuracy: 0.9566 - auc: 0.9793 - val_loss: 0.
4428 - val_tp: 4338892.0000 - val_fp: 35099.0000 - val_tn: 20239610.0000 - val_fn: 1600799.0000 - val_precision: 0.9920 - val_recall: 0.7305 - val_accuracy: 0.9376 - val_auc: 0.9713
Epoch 14/100
1280/1280 [==============================] - 1957s 2s/step - loss: 0.2883 - tp: 21882504.0000 - fp: 2239269.0000 - tn: 78431648.0000 - fn: 2304140.0000 - precision: 0.9072 - recall: 0.9047 - accuracy: 0.9567 - auc: 0.9796 - val_loss: 0.
2957 - val_tp: 5289461.0000 - val_fp: 449961.0000 - val_tn: 19835942.0000 - val_fn: 639032.0000 - val_precision: 0.9216 - val_recall: 0.8922 - val_accuracy: 0.9585 - val_auc: 0.9820
Epoch 15/100
1280/1280 [==============================] - 2044s 2s/step - loss: 0.2825 - tp: 21866824.0000 - fp: 2177254.0000 - tn: 78541248.0000 - fn: 2272316.0000 - precision: 0.9094 - recall: 0.9059 - accuracy: 0.9576 - auc: 0.9802 - val_loss: 0.
3209 - val_tp: 5233737.0000 - val_fp: 520949.0000 - val_tn: 19784424.0000 - val_fn: 675290.0000 - val_precision: 0.9095 - val_recall: 0.8857 - val_accuracy: 0.9544 - val_auc: 0.9800
Epoch 16/100
1280/1280 [==============================] - 1957s 2s/step - loss: 0.2769 - tp: 21749612.0000 - fp: 2170557.0000 - tn: 78722872.0000 - fn: 2214598.0000 - precision: 0.9093 - recall: 0.9076 - accuracy: 0.9582 - auc: 0.9802 - val_loss: 0.
3247 - val_tp: 5660332.0000 - val_fp: 828589.0000 - val_tn: 19303372.0000 - val_fn: 422111.0000 - val_precision: 0.8723 - val_recall: 0.9306 - val_accuracy: 0.9523 - val_auc: 0.9859
Epoch 17/100
1280/1280 [==============================] - 1949s 2s/step - loss: 0.2774 - tp: 21816756.0000 - fp: 2164372.0000 - tn: 78635128.0000 - fn: 2241237.0000 - precision: 0.9097 - recall: 0.9068 - accuracy: 0.9580 - auc: 0.9806 - val_loss: 0.
3148 - val_tp: 4953313.0000 - val_fp: 118813.0000 - val_tn: 20032204.0000 - val_fn: 1110073.0000 - val_precision: 0.9766 - val_recall: 0.8169 - val_accuracy: 0.9531 - val_auc: 0.9788
Epoch 18/100
1280/1280 [==============================] - 1944s 2s/step - loss: 0.2725 - tp: 21678578.0000 - fp: 2114671.0000 - tn: 78865952.0000 - fn: 2198422.0000 - precision: 0.9111 - recall: 0.9079 - accuracy: 0.9589 - auc: 0.9807 - val_loss: 0.
3363 - val_tp: 4425577.0000 - val_fp: 55980.0000 - val_tn: 20363536.0000 - val_fn: 1369309.0000 - val_precision: 0.9875 - val_recall: 0.7637 - val_accuracy: 0.9456 - val_auc: 0.9739
Epoch 19/100
1280/1280 [==============================] - 1947s 2s/step - loss: 0.2757 - tp: 22040606.0000 - fp: 2177952.0000 - tn: 78420008.0000 - fn: 2218997.0000 - precision: 0.9101 - recall: 0.9085 - accuracy: 0.9581 - auc: 0.9805 - val_loss: 0.
3061 - val_tp: 5376668.0000 - val_fp: 295874.0000 - val_tn: 19715332.0000 - val_fn: 826533.0000 - val_precision: 0.9478 - val_recall: 0.8668 - val_accuracy: 0.9572 - val_auc: 0.9796
Epoch 20/100
1280/1280 [==============================] - 1953s 2s/step - loss: 0.2758 - tp: 21804466.0000 - fp: 2129789.0000 - tn: 78704432.0000 - fn: 2218904.0000 - precision: 0.9110 - recall: 0.9076 - accuracy: 0.9585 - auc: 0.9805 - val_loss: 0.
3864 - val_tp: 4404185.0000 - val_fp: 40930.0000 - val_tn: 20226700.0000 - val_fn: 1542578.0000 - val_precision: 0.9908 - val_recall: 0.7406 - val_accuracy: 0.9396 - val_auc: 0.9670
Epoch 21/100
1280/1280 [==============================] - 1956s 2s/step - loss: 0.2715 - tp: 21833378.0000 - fp: 2128388.0000 - tn: 78697688.0000 - fn: 2198099.0000 - precision: 0.9112 - recall: 0.9085 - accuracy: 0.9587 - auc: 0.9812 - val_loss: 0.
3215 - val_tp: 4933923.0000 - val_fp: 100047.0000 - val_tn: 20044032.0000 - val_fn: 1136392.0000 - val_precision: 0.9801 - val_recall: 0.8128 - val_accuracy: 0.9528 - val_auc: 0.9780
Epoch 22/100
1280/1280 [==============================] - 1957s 2s/step - loss: 0.2685 - tp: 22131982.0000 - fp: 2123964.0000 - tn: 78455072.0000 - fn: 2146571.0000 - precision: 0.9124 - recall: 0.9116 - accuracy: 0.9593 - auc: 0.9811 - val_loss: 0.
2843 - val_tp: 5065657.0000 - val_fp: 196579.0000 - val_tn: 20093300.0000 - val_fn: 858874.0000 - val_precision: 0.9626 - val_recall: 0.8550 - val_accuracy: 0.9597 - val_auc: 0.9806
Epoch 23/100
1280/1280 [==============================] - 1974s 2s/step - loss: 0.2667 - tp: 21803614.0000 - fp: 2090465.0000 - tn: 78828520.0000 - fn: 2134989.0000 - precision: 0.9125 - recall: 0.9108 - accuracy: 0.9597 - auc: 0.9811 - val_loss: 0.
3048 - val_tp: 5024525.0000 - val_fp: 147570.0000 - val_tn: 20040236.0000 - val_fn: 1002069.0000 - val_precision: 0.9715 - val_recall: 0.8337 - val_accuracy: 0.9561 - val_auc: 0.9796
Epoch 24/100
1280/1280 [==============================] - 1969s 2s/step - loss: 0.2636 - tp: 21975042.0000 - fp: 2062614.0000 - tn: 78687080.0000 - fn: 2132792.0000 - precision: 0.9142 - recall: 0.9115 - accuracy: 0.9600 - auc: 0.9817 - val_loss: 0.
2889 - val_tp: 5170852.0000 - val_fp: 206375.0000 - val_tn: 19954888.0000 - val_fn: 882289.0000 - val_precision: 0.9616 - val_recall: 0.8542 - val_accuracy: 0.9585 - val_auc: 0.9811
Epoch 25/100
1280/1280 [==============================] - 1966s 2s/step - loss: 0.2654 - tp: 21864694.0000 - fp: 2102611.0000 - tn: 78755144.0000 - fn: 2135188.0000 - precision: 0.9123 - recall: 0.9110 - accuracy: 0.9596 - auc: 0.9815 - val_loss: 0.
2742 - val_tp: 5232587.0000 - val_fp: 220712.0000 - val_tn: 19933466.0000 - val_fn: 827639.0000 - val_precision: 0.9595 - val_recall: 0.8634 - val_accuracy: 0.9600 - val_auc: 0.9812
Epoch 26/100
1280/1280 [==============================] - 1968s 2s/step - loss: 0.2665 - tp: 21870048.0000 - fp: 2112301.0000 - tn: 78741728.0000 - fn: 2133559.0000 - precision: 0.9119 - recall: 0.9111 - accuracy: 0.9595 - auc: 0.9816 - val_loss: 0.
3169 - val_tp: 4986570.0000 - val_fp: 130988.0000 - val_tn: 20053542.0000 - val_fn: 1043297.0000 - val_precision: 0.9744 - val_recall: 0.8270 - val_accuracy: 0.9552 - val_auc: 0.9802
Epoch 27/100
1280/1280 [==============================] - 1952s 2s/step - loss: 0.2678 - tp: 21893164.0000 - fp: 2085167.0000 - tn: 78709264.0000 - fn: 2170034.0000 - precision: 0.9130 - recall: 0.9098 - accuracy: 0.9594 - auc: 0.9813 - val_loss: 0.
2900 - val_tp: 5054986.0000 - val_fp: 200466.0000 - val_tn: 20084900.0000 - val_fn: 874040.0000 - val_precision: 0.9619 - val_recall: 0.8526 - val_accuracy: 0.9590 - val_auc: 0.9804
Epoch 28/100
1280/1280 [==============================] - 1958s 2s/step - loss: 0.2650 - tp: 22049098.0000 - fp: 2106978.0000 - tn: 78571512.0000 - fn: 2130046.0000 - precision: 0.9128 - recall: 0.9119 - accuracy: 0.9596 - auc: 0.9817 - val_loss: 0.
2835 - val_tp: 4962208.0000 - val_fp: 180333.0000 - val_tn: 20181980.0000 - val_fn: 889878.0000 - val_precision: 0.9649 - val_recall: 0.8479 - val_accuracy: 0.9592 - val_auc: 0.9790
Epoch 29/100
1280/1280 [==============================] - 1966s 2s/step - loss: 0.2624 - tp: 21985904.0000 - fp: 2088019.0000 - tn: 78675856.0000 - fn: 2107816.0000 - precision: 0.9133 - recall: 0.9125 - accuracy: 0.9600 - auc: 0.9818 - val_loss: 0.
2769 - val_tp: 5178520.0000 - val_fp: 252398.0000 - val_tn: 20002776.0000 - val_fn: 780712.0000 - val_precision: 0.9535 - val_recall: 0.8690 - val_accuracy: 0.9606 - val_auc: 0.9797
Epoch 30/100
1280/1280 [==============================] - 1938s 2s/step - loss: 0.2616 - tp: 21893676.0000 - fp: 2096355.0000 - tn: 78765992.0000 - fn: 2101589.0000 - precision: 0.9126 - recall: 0.9124 - accuracy: 0.9600 - auc: 0.9820 - val_loss: 0.
2931 - val_tp: 5091699.0000 - val_fp: 180837.0000 - val_tn: 20020972.0000 - val_fn: 920891.0000 - val_precision: 0.9657 - val_recall: 0.8468 - val_accuracy: 0.9580 - val_auc: 0.9803
Epoch 31/100
1280/1280 [==============================] - 1932s 2s/step - loss: 0.2629 - tp: 21900190.0000 - fp: 2102823.0000 - tn: 78738360.0000 - fn: 2116254.0000 - precision: 0.9124 - recall: 0.9119 - accuracy: 0.9598 - auc: 0.9818 - val_loss: 0.
2833 - val_tp: 5068565.0000 - val_fp: 190985.0000 - val_tn: 20068752.0000 - val_fn: 886105.0000 - val_precision: 0.9637 - val_recall: 0.8512 - val_accuracy: 0.9589 - val_auc: 0.9797
Epoch 32/100
1280/1280 [==============================] - 1936s 2s/step - loss: 0.2656 - tp: 21855648.0000 - fp: 2070417.0000 - tn: 78781648.0000 - fn: 2149897.0000 - precision: 0.9135 - recall: 0.9104 - accuracy: 0.9598 - auc: 0.9816 - val_loss: 0.
3086 - val_tp: 5081221.0000 - val_fp: 145540.0000 - val_tn: 20010884.0000 - val_fn: 976755.0000 - val_precision: 0.9722 - val_recall: 0.8388 - val_accuracy: 0.9572 - val_auc: 0.9796
Epoch 33/100
1280/1280 [==============================] - 1896s 1s/step - loss: 0.2601 - tp: 21847706.0000 - fp: 2017783.0000 - tn: 78866344.0000 - fn: 2125796.0000 - precision: 0.9155 - recall: 0.9113 - accuracy: 0.9605 - auc: 0.9820 - val_loss: 0.
3225 - val_tp: 5136058.0000 - val_fp: 129931.0000 - val_tn: 19900672.0000 - val_fn: 1047735.0000 - val_precision: 0.9753 - val_recall: 0.8306 - val_accuracy: 0.9551 - val_auc: 0.9803
Epoch 34/100
1280/1280 [==============================] - 1883s 1s/step - loss: 0.2647 - tp: 21866856.0000 - fp: 2063199.0000 - tn: 78792768.0000 - fn: 2134767.0000 - precision: 0.9138 - recall: 0.9111 - accuracy: 0.9600 - auc: 0.9815 - val_loss: 0.
2877 - val_tp: 5081384.0000 - val_fp: 183837.0000 - val_tn: 20092414.0000 - val_fn: 856762.0000 - val_precision: 0.9651 - val_recall: 0.8557 - val_accuracy: 0.9603 - val_auc: 0.9819
Epoch 35/100
1280/1280 [==============================] - 1909s 1s/step - loss: 0.2644 - tp: 21793352.0000 - fp: 2043704.0000 - tn: 78858192.0000 - fn: 2162348.0000 - precision: 0.9143 - recall: 0.9097 - accuracy: 0.9599 - auc: 0.9813 - val_loss: 0.
2804 - val_tp: 5229124.0000 - val_fp: 199996.0000 - val_tn: 19913378.0000 - val_fn: 871912.0000 - val_precision: 0.9632 - val_recall: 0.8571 - val_accuracy: 0.9591 - val_auc: 0.9811
Epoch 36/100
1280/1280 [==============================] - 1926s 2s/step - loss: 0.2624 - tp: 21694132.0000 - fp: 2078119.0000 - tn: 78982728.0000 - fn: 2102522.0000 - precision: 0.9126 - recall: 0.9116 - accuracy: 0.9601 - auc: 0.9816 - val_loss: 0.
2822 - val_tp: 5091321.0000 - val_fp: 163063.0000 - val_tn: 20040276.0000 - val_fn: 919745.0000 - val_precision: 0.9690 - val_recall: 0.8470 - val_accuracy: 0.9587 - val_auc: 0.9807
Epoch 37/100
1280/1280 [==============================] - 1904s 1s/step - loss: 0.2629 - tp: 21760348.0000 - fp: 2034859.0000 - tn: 78921984.0000 - fn: 2140485.0000 - precision: 0.9145 - recall: 0.9104 - accuracy: 0.9602 - auc: 0.9816 - val_loss: 0.
3176 - val_tp: 5181435.0000 - val_fp: 158434.0000 - val_tn: 19879032.0000 - val_fn: 995505.0000 - val_precision: 0.9703 - val_recall: 0.8388 - val_accuracy: 0.9560 - val_auc: 0.9790
Epoch 38/100
1280/1280 [==============================] - 1894s 1s/step - loss: 0.2660 - tp: 21873788.0000 - fp: 2074315.0000 - tn: 78773216.0000 - fn: 2136309.0000 - precision: 0.9134 - recall: 0.9110 - accuracy: 0.9598 - auc: 0.9814 - val_loss: 0.
2881 - val_tp: 4927277.0000 - val_fp: 173887.0000 - val_tn: 20220536.0000 - val_fn: 892705.0000 - val_precision: 0.9659 - val_recall: 0.8466 - val_accuracy: 0.9593 - val_auc: 0.9789
Epoch 39/100
1280/1280 [==============================] - 1899s 1s/step - loss: 0.2622 - tp: 21838440.0000 - fp: 2067724.0000 - tn: 78836336.0000 - fn: 2115209.0000 - precision: 0.9135 - recall: 0.9117 - accuracy: 0.9601 - auc: 0.9818 - val_loss: 0.
2978 - val_tp: 4918815.0000 - val_fp: 143772.0000 - val_tn: 20190378.0000 - val_fn: 961431.0000 - val_precision: 0.9716 - val_recall: 0.8365 - val_accuracy: 0.9578 - val_auc: 0.9807
Epoch 40/100
1280/1280 [==============================] - 1888s 1s/step - loss: 0.2648 - tp: 21872584.0000 - fp: 2050789.0000 - tn: 78773392.0000 - fn: 2160846.0000 - precision: 0.9143 - recall: 0.9101 - accuracy: 0.9598 - auc: 0.9816 - val_loss: 0.
2909 - val_tp: 5274649.0000 - val_fp: 191573.0000 - val_tn: 19822700.0000 - val_fn: 925468.0000 - val_precision: 0.9650 - val_recall: 0.8507 - val_accuracy: 0.9574 - val_auc: 0.9799
Epoch 41/100
1280/1280 [==============================] - 1910s 1s/step - loss: 0.2648 - tp: 21696836.0000 - fp: 2093251.0000 - tn: 78944688.0000 - fn: 2122855.0000 - precision: 0.9120 - recall: 0.9109 - accuracy: 0.9598 - auc: 0.9814 - val_loss: 0.
2811 - val_tp: 5266549.0000 - val_fp: 194645.0000 - val_tn: 19876088.0000 - val_fn: 877120.0000 - val_precision: 0.9644 - val_recall: 0.8572 - val_accuracy: 0.9591 - val_auc: 0.9805
Epoch 42/100
1280/1280 [==============================] - 1912s 1s/step - loss: 0.2653 - tp: 21816640.0000 - fp: 2044165.0000 - tn: 78832952.0000 - fn: 2163912.0000 - precision: 0.9143 - recall: 0.9098 - accuracy: 0.9599 - auc: 0.9813 - val_loss: 0.
2983 - val_tp: 4907063.0000 - val_fp: 162204.0000 - val_tn: 20208222.0000 - val_fn: 936914.0000 - val_precision: 0.9680 - val_recall: 0.8397 - val_accuracy: 0.9581 - val_auc: 0.9799
Epoch 43/100
1280/1280 [==============================] - 1898s 1s/step - loss: 0.2646 - tp: 22222472.0000 - fp: 2064761.0000 - tn: 78412904.0000 - fn: 2157526.0000 - precision: 0.9150 - recall: 0.9115 - accuracy: 0.9597 - auc: 0.9817 - val_loss: 0.
2900 - val_tp: 5266505.0000 - val_fp: 189111.0000 - val_tn: 19834046.0000 - val_fn: 924743.0000 - val_precision: 0.9653 - val_recall: 0.8506 - val_accuracy: 0.9575 - val_auc: 0.9807
Epoch 44/100
1280/1280 [==============================] - 1937s 2s/step - loss: 0.2625 - tp: 21883992.0000 - fp: 2040394.0000 - tn: 78792808.0000 - fn: 2140429.0000 - precision: 0.9147 - recall: 0.9109 - accuracy: 0.9601 - auc: 0.9817 - val_loss: 0.
3128 - val_tp: 4872292.0000 - val_fp: 140248.0000 - val_tn: 20261372.0000 - val_fn: 940479.0000 - val_precision: 0.9720 - val_recall: 0.8382 - val_accuracy: 0.9588 - val_auc: 0.9805
Epoch 45/100
1280/1280 [==============================] - 1931s 2s/step - loss: 0.2641 - tp: 21804156.0000 - fp: 2047906.0000 - tn: 78877472.0000 - fn: 2128092.0000 - precision: 0.9141 - recall: 0.9111 - accuracy: 0.9602 - auc: 0.9814 - val_loss: 0.
2983 - val_tp: 5262537.0000 - val_fp: 177583.0000 - val_tn: 19827930.0000 - val_fn: 946346.0000 - val_precision: 0.9674 - val_recall: 0.8476 - val_accuracy: 0.9571 - val_auc: 0.9792
Epoch 46/100
1280/1280 [==============================] - 1941s 2s/step - loss: 0.2640 - tp: 21923374.0000 - fp: 2080069.0000 - tn: 78717264.0000 - fn: 2136918.0000 - precision: 0.9133 - recall: 0.9112 - accuracy: 0.9598 - auc: 0.9816 - val_loss: 0.
2822 - val_tp: 5087710.0000 - val_fp: 182249.0000 - val_tn: 20063030.0000 - val_fn: 881411.0000 - val_precision: 0.9654 - val_recall: 0.8523 - val_accuracy: 0.9594 - val_auc: 0.9803
Epoch 47/100
1280/1280 [==============================] - 1944s 2s/step - loss: 0.2620 - tp: 21961636.0000 - fp: 2068701.0000 - tn: 78701112.0000 - fn: 2126225.0000 - precision: 0.9139 - recall: 0.9117 - accuracy: 0.9600 - auc: 0.9818 - val_loss: 0.
2982 - val_tp: 5201830.0000 - val_fp: 170225.0000 - val_tn: 19876590.0000 - val_fn: 965741.0000 - val_precision: 0.9683 - val_recall: 0.8434 - val_accuracy: 0.9567 - val_auc: 0.9799
Epoch 48/100
1280/1280 [==============================] - 1948s 2s/step - loss: 0.2631 - tp: 21971392.0000 - fp: 2056324.0000 - tn: 78695536.0000 - fn: 2134300.0000 - precision: 0.9144 - recall: 0.9115 - accuracy: 0.9600 - auc: 0.9818 - val_loss: 0.
2906 - val_tp: 5083102.0000 - val_fp: 170364.0000 - val_tn: 20046178.0000 - val_fn: 914747.0000 - val_precision: 0.9676 - val_recall: 0.8475 - val_accuracy: 0.9586 - val_auc: 0.9792
Epoch 49/100
1280/1280 [==============================] - 1937s 2s/step - loss: 0.2629 - tp: 21911328.0000 - fp: 2071634.0000 - tn: 78755200.0000 - fn: 2119426.0000 - precision: 0.9136 - recall: 0.9118 - accuracy: 0.9600 - auc: 0.9818 - val_loss: 0.
2839 - val_tp: 5050322.0000 - val_fp: 183661.0000 - val_tn: 20111232.0000 - val_fn: 869193.0000 - val_precision: 0.9649 - val_recall: 0.8532 - val_accuracy: 0.9598 - val_auc: 0.9806
Epoch 50/100
1280/1280 [==============================] - 1964s 2s/step - loss: 0.2634 - tp: 21943356.0000 - fp: 2042601.0000 - tn: 78728936.0000 - fn: 2142696.0000 - precision: 0.9148 - recall: 0.9110 - accuracy: 0.9601 - auc: 0.9816 - val_loss: 0.
2906 - val_tp: 5176414.0000 - val_fp: 195927.0000 - val_tn: 19950280.0000 - val_fn: 891781.0000 - val_precision: 0.9635 - val_recall: 0.8530 - val_accuracy: 0.9585 - val_auc: 0.9800
Epoch 51/100
1280/1280 [==============================] - 1949s 2s/step - loss: 0.2619 - tp: 21905776.0000 - fp: 2054187.0000 - tn: 78790400.0000 - fn: 2107240.0000 - precision: 0.9143 - recall: 0.9122 - accuracy: 0.9603 - auc: 0.9818 - val_loss: 0.
2845 - val_tp: 5128164.0000 - val_fp: 168438.0000 - val_tn: 19986128.0000 - val_fn: 931667.0000 - val_precision: 0.9682 - val_recall: 0.8463 - val_accuracy: 0.9580 - val_auc: 0.9810
Epoch 52/100
1280/1280 [==============================] - 1959s 2s/step - loss: 0.2638 - tp: 21870924.0000 - fp: 2087711.0000 - tn: 78782424.0000 - fn: 2116607.0000 - precision: 0.9129 - recall: 0.9118 - accuracy: 0.9599 - auc: 0.9817 - val_loss: 0.
3062 - val_tp: 5273162.0000 - val_fp: 169306.0000 - val_tn: 19832368.0000 - val_fn: 939566.0000 - val_precision: 0.9689 - val_recall: 0.8488 - val_accuracy: 0.9577 - val_auc: 0.9815
Epoch 53/100
1280/1280 [==============================] - 1958s 2s/step - loss: 0.2648 - tp: 21947556.0000 - fp: 2065779.0000 - tn: 78702432.0000 - fn: 2141837.0000 - precision: 0.9140 - recall: 0.9111 - accuracy: 0.9599 - auc: 0.9816 - val_loss: 0.
2945 - val_tp: 5204859.0000 - val_fp: 153095.0000 - val_tn: 19890892.0000 - val_fn: 965553.0000 - val_precision: 0.9714 - val_recall: 0.8435 - val_accuracy: 0.9573 - val_auc: 0.9800
Epoch 54/100
1280/1280 [==============================] - 1937s 2s/step - loss: 0.2635 - tp: 21905812.0000 - fp: 2049643.0000 - tn: 78761584.0000 - fn: 2140551.0000 - precision: 0.9144 - recall: 0.9110 - accuracy: 0.9600 - auc: 0.9816 - val_loss: 0.
3064 - val_tp: 5062076.0000 - val_fp: 149992.0000 - val_tn: 20054216.0000 - val_fn: 948115.0000 - val_precision: 0.9712 - val_recall: 0.8422 - val_accuracy: 0.9581 - val_auc: 0.9815
Epoch 55/100
1280/1280 [==============================] - 1958s 2s/step - loss: 0.2653 - tp: 22046804.0000 - fp: 2122362.0000 - tn: 78569416.0000 - fn: 2119021.0000 - precision: 0.9122 - recall: 0.9123 - accuracy: 0.9596 - auc: 0.9818 - val_loss: 0.
2913 - val_tp: 5030327.0000 - val_fp: 162722.0000 - val_tn: 20130282.0000 - val_fn: 891077.0000 - val_precision: 0.9687 - val_recall: 0.8495 - val_accuracy: 0.9598 - val_auc: 0.9812
Epoch 56/100
1280/1280 [==============================] - 1946s 2s/step - loss: 0.2644 - tp: 21869356.0000 - fp: 2060246.0000 - tn: 78783952.0000 - fn: 2144064.0000 - precision: 0.9139 - recall: 0.9107 - accuracy: 0.9599 - auc: 0.9814 - val_loss: 0.
2816 - val_tp: 5105422.0000 - val_fp: 167571.0000 - val_tn: 20081292.0000 - val_fn: 860105.0000 - val_precision: 0.9682 - val_recall: 0.8558 - val_accuracy: 0.9608 - val_auc: 0.9805
Epoch 57/100
1280/1280 [==============================] - 1921s 2s/step - loss: 0.2642 - tp: 22001298.0000 - fp: 2070341.0000 - tn: 78641616.0000 - fn: 2144355.0000 - precision: 0.9140 - recall: 0.9112 - accuracy: 0.9598 - auc: 0.9816 - val_loss: 0.
3304 - val_tp: 4955272.0000 - val_fp: 133108.0000 - val_tn: 20129192.0000 - val_fn: 996829.0000 - val_precision: 0.9738 - val_recall: 0.8325 - val_accuracy: 0.9569 - val_auc: 0.9802
Epoch 58/100
1280/1280 [==============================] - 1919s 1s/step - loss: 0.2644 - tp: 21756178.0000 - fp: 2097986.0000 - tn: 78861704.0000 - fn: 2141728.0000 - precision: 0.9120 - recall: 0.9104 - accuracy: 0.9596 - auc: 0.9815 - val_loss: 0.
2800 - val_tp: 5173984.0000 - val_fp: 172737.0000 - val_tn: 19995672.0000 - val_fn: 872006.0000 - val_precision: 0.9677 - val_recall: 0.8558 - val_accuracy: 0.9601 - val_auc: 0.9811
Epoch 59/100
1280/1280 [==============================] - 1937s 2s/step - loss: 0.2639 - tp: 21965856.0000 - fp: 2104414.0000 - tn: 78670312.0000 - fn: 2117059.0000 - precision: 0.9126 - recall: 0.9121 - accuracy: 0.9597 - auc: 0.9817 - val_loss: 0.
2969 - val_tp: 5288252.0000 - val_fp: 187540.0000 - val_tn: 19831520.0000 - val_fn: 907092.0000 - val_precision: 0.9658 - val_recall: 0.8536 - val_accuracy: 0.9582 - val_auc: 0.9792
Epoch 60/100
1280/1280 [==============================] - 1942s 2s/step - loss: 0.2652 - tp: 21972316.0000 - fp: 2103459.0000 - tn: 78666280.0000 - fn: 2115554.0000 - precision: 0.9126 - recall: 0.9122 - accuracy: 0.9598 - auc: 0.9817 - val_loss: 0.
2818 - val_tp: 5166245.0000 - val_fp: 190846.0000 - val_tn: 19971494.0000 - val_fn: 885813.0000 - val_precision: 0.9644 - val_recall: 0.8536 - val_accuracy: 0.9589 - val_auc: 0.9810
Epoch 61/100
1280/1280 [==============================] - 1925s 2s/step - loss: 0.2651 - tp: 21878652.0000 - fp: 2066501.0000 - tn: 78782176.0000 - fn: 2130193.0000 - precision: 0.9137 - recall: 0.9113 - accuracy: 0.9600 - auc: 0.9814 - val_loss: 0.
2793 - val_tp: 5085499.0000 - val_fp: 194718.0000 - val_tn: 20060124.0000 - val_fn: 874059.0000 - val_precision: 0.9631 - val_recall: 0.8533 - val_accuracy: 0.9592 - val_auc: 0.9805
Epoch 62/100
1280/1280 [==============================] - 1922s 2s/step - loss: 0.2645 - tp: 21871498.0000 - fp: 2095977.0000 - tn: 78758960.0000 - fn: 2131121.0000 - precision: 0.9125 - recall: 0.9112 - accuracy: 0.9597 - auc: 0.9816 - val_loss: 0.
3215 - val_tp: 5111082.0000 - val_fp: 129336.0000 - val_tn: 19952116.0000 - val_fn: 1021864.0000 - val_precision: 0.9753 - val_recall: 0.8334 - val_accuracy: 0.9561 - val_auc: 0.9808
Epoch 63/100
1280/1280 [==============================] - 1938s 2s/step - loss: 0.2653 - tp: 21667542.0000 - fp: 2040165.0000 - tn: 78980168.0000 - fn: 2169717.0000 - precision: 0.9139 - recall: 0.9090 - accuracy: 0.9599 - auc: 0.9811 - val_loss: 0.
2935 - val_tp: 5105651.0000 - val_fp: 157582.0000 - val_tn: 19997984.0000 - val_fn: 953187.0000 - val_precision: 0.9701 - val_recall: 0.8427 - val_accuracy: 0.9576 - val_auc: 0.9795
Epoch 64/100
1280/1280 [==============================] - 1922s 2s/step - loss: 0.2621 - tp: 21566260.0000 - fp: 2041653.0000 - tn: 79126792.0000 - fn: 2122825.0000 - precision: 0.9135 - recall: 0.9104 - accuracy: 0.9603 - auc: 0.9816 - val_loss: 0.
2868 - val_tp: 5295960.0000 - val_fp: 181316.0000 - val_tn: 19817780.0000 - val_fn: 919349.0000 - val_precision: 0.9669 - val_recall: 0.8521 - val_accuracy: 0.9580 - val_auc: 0.9802
Epoch 65/100
1280/1280 [==============================] - 1930s 2s/step - loss: 0.2618 - tp: 21803380.0000 - fp: 2046884.0000 - tn: 78875232.0000 - fn: 2132114.0000 - precision: 0.9142 - recall: 0.9109 - accuracy: 0.9601 - auc: 0.9818 - val_loss: 0.
2655 - val_tp: 4998172.0000 - val_fp: 220002.0000 - val_tn: 20208396.0000 - val_fn: 787827.0000 - val_precision: 0.9578 - val_recall: 0.8638 - val_accuracy: 0.9616 - val_auc: 0.9810
Epoch 66/100
1280/1280 [==============================] - 1940s 2s/step - loss: 0.2621 - tp: 21715956.0000 - fp: 2060856.0000 - tn: 78976016.0000 - fn: 2104835.0000 - precision: 0.9133 - recall: 0.9116 - accuracy: 0.9603 - auc: 0.9817 - val_loss: 0.
2934 - val_tp: 4858098.0000 - val_fp: 162111.0000 - val_tn: 20270832.0000 - val_fn: 923350.0000 - val_precision: 0.9677 - val_recall: 0.8403 - val_accuracy: 0.9586 - val_auc: 0.9797
Epoch 67/100
1280/1280 [==============================] - 1936s 2s/step - loss: 0.2641 - tp: 22101508.0000 - fp: 2096796.0000 - tn: 78514056.0000 - fn: 2145167.0000 - precision: 0.9133 - recall: 0.9115 - accuracy: 0.9595 - auc: 0.9819 - val_loss: 0.
2935 - val_tp: 5311098.0000 - val_fp: 189758.0000 - val_tn: 19779840.0000 - val_fn: 933704.0000 - val_precision: 0.9655 - val_recall: 0.8505 - val_accuracy: 0.9571 - val_auc: 0.9804
Epoch 68/100
1280/1280 [==============================] - 1929s 2s/step - loss: 0.2639 - tp: 21973892.0000 - fp: 2058296.0000 - tn: 78690816.0000 - fn: 2134585.0000 - precision: 0.9144 - recall: 0.9115 - accuracy: 0.9600 - auc: 0.9816 - val_loss: 0.
3021 - val_tp: 5393303.0000 - val_fp: 200762.0000 - val_tn: 19679066.0000 - val_fn: 941275.0000 - val_precision: 0.9641 - val_recall: 0.8514 - val_accuracy: 0.9564 - val_auc: 0.9795
Epoch 69/100
1280/1280 [==============================] - 1928s 2s/step - loss: 0.2632 - tp: 21797492.0000 - fp: 2054212.0000 - tn: 78874840.0000 - fn: 2131042.0000 - precision: 0.9139 - recall: 0.9109 - accuracy: 0.9601 - auc: 0.9816 - val_loss: 0.
2970 - val_tp: 5102175.0000 - val_fp: 191344.0000 - val_tn: 19995848.0000 - val_fn: 925037.0000 - val_precision: 0.9639 - val_recall: 0.8465 - val_accuracy: 0.9574 - val_auc: 0.9797
Epoch 70/100
1280/1280 [==============================] - 1936s 2s/step - loss: 0.2629 - tp: 21925510.0000 - fp: 2063262.0000 - tn: 78733408.0000 - fn: 2135445.0000 - precision: 0.9140 - recall: 0.9112 - accuracy: 0.9600 - auc: 0.9818 - val_loss: 0.
2835 - val_tp: 5162856.0000 - val_fp: 228043.0000 - val_tn: 19969488.0000 - val_fn: 854014.0000 - val_precision: 0.9577 - val_recall: 0.8581 - val_accuracy: 0.9587 - val_auc: 0.9802
Epoch 71/100
1280/1280 [==============================] - 1930s 2s/step - loss: 0.2606 - tp: 21986884.0000 - fp: 2118880.0000 - tn: 78684536.0000 - fn: 2067241.0000 - precision: 0.9121 - recall: 0.9141 - accuracy: 0.9601 - auc: 0.9822 - val_loss: 0.
2809 - val_tp: 5176839.0000 - val_fp: 235191.0000 - val_tn: 19977128.0000 - val_fn: 825244.0000 - val_precision: 0.9565 - val_recall: 0.8625 - val_accuracy: 0.9595 - val_auc: 0.9812
Epoch 72/100
1280/1280 [==============================] - 1952s 2s/step - loss: 0.2618 - tp: 22038310.0000 - fp: 2056830.0000 - tn: 78655040.0000 - fn: 2107382.0000 - precision: 0.9146 - recall: 0.9127 - accuracy: 0.9603 - auc: 0.9820 - val_loss: 0.
2895 - val_tp: 5367415.0000 - val_fp: 210844.0000 - val_tn: 19760872.0000 - val_fn: 875261.0000 - val_precision: 0.9622 - val_recall: 0.8598 - val_accuracy: 0.9586 - val_auc: 0.9801
Epoch 73/100
1280/1280 [==============================] - 1931s 2s/step - loss: 0.2642 - tp: 22040244.0000 - fp: 2079148.0000 - tn: 78597176.0000 - fn: 2140976.0000 - precision: 0.9138 - recall: 0.9115 - accuracy: 0.9598 - auc: 0.9817 - val_loss: 0.
3093 - val_tp: 5018548.0000 - val_fp: 149682.0000 - val_tn: 20096504.0000 - val_fn: 949668.0000 - val_precision: 0.9710 - val_recall: 0.8409 - val_accuracy: 0.9581 - val_auc: 0.9804
Epoch 74/100
1280/1280 [==============================] - 1891s 1s/step - loss: 0.2648 - tp: 21823536.0000 - fp: 2026553.0000 - tn: 78842960.0000 - fn: 2164550.0000 - precision: 0.9150 - recall: 0.9098 - accuracy: 0.9600 - auc: 0.9815 - val_loss: 0.
2941 - val_tp: 5138005.0000 - val_fp: 183811.0000 - val_tn: 19970134.0000 - val_fn: 922446.0000 - val_precision: 0.9655 - val_recall: 0.8478 - val_accuracy: 0.9578 - val_auc: 0.9795
Epoch 75/100
1280/1280 [==============================] - 1896s 1s/step - loss: 0.2599 - tp: 21797956.0000 - fp: 2051529.0000 - tn: 78902288.0000 - fn: 2105822.0000 - precision: 0.9140 - recall: 0.9119 - accuracy: 0.9604 - auc: 0.9820 - val_loss: 0.
2866 - val_tp: 5237317.0000 - val_fp: 179478.0000 - val_tn: 19883768.0000 - val_fn: 913840.0000 - val_precision: 0.9669 - val_recall: 0.8514 - val_accuracy: 0.9583 - val_auc: 0.9806
Epoch 76/100
1280/1280 [==============================] - 1920s 2s/step - loss: 0.2644 - tp: 21883500.0000 - fp: 2043967.0000 - tn: 78765848.0000 - fn: 2164260.0000 - precision: 0.9146 - recall: 0.9100 - accuracy: 0.9599 - auc: 0.9815 - val_loss: 0.
2969 - val_tp: 4856771.0000 - val_fp: 158534.0000 - val_tn: 20318780.0000 - val_fn: 880306.0000 - val_precision: 0.9684 - val_recall: 0.8466 - val_accuracy: 0.9604 - val_auc: 0.9810
Epoch 77/100
1280/1280 [==============================] - 1950s 2s/step - loss: 0.2649 - tp: 21977528.0000 - fp: 2098101.0000 - tn: 78653504.0000 - fn: 2128494.0000 - precision: 0.9129 - recall: 0.9117 - accuracy: 0.9597 - auc: 0.9816 - val_loss: 0.
2794 - val_tp: 5248088.0000 - val_fp: 201804.0000 - val_tn: 19923706.0000 - val_fn: 840800.0000 - val_precision: 0.9630 - val_recall: 0.8619 - val_accuracy: 0.9602 - val_auc: 0.9806
Epoch 78/100
1280/1280 [==============================] - 1951s 2s/step - loss: 0.2630 - tp: 21619006.0000 - fp: 2054910.0000 - tn: 79065992.0000 - fn: 2117649.0000 - precision: 0.9132 - recall: 0.9108 - accuracy: 0.9602 - auc: 0.9815 - val_loss: 0.
2924 - val_tp: 5121101.0000 - val_fp: 170363.0000 - val_tn: 20022854.0000 - val_fn: 900073.0000 - val_precision: 0.9678 - val_recall: 0.8505 - val_accuracy: 0.9592 - val_auc: 0.9811
Epoch 79/100
1280/1280 [==============================] - 1956s 2s/step - loss: 0.2646 - tp: 21744384.0000 - fp: 2060624.0000 - tn: 78899240.0000 - fn: 2153398.0000 - precision: 0.9134 - recall: 0.9099 - accuracy: 0.9598 - auc: 0.9814 - val_loss: 0.
2982 - val_tp: 4993843.0000 - val_fp: 185702.0000 - val_tn: 20120108.0000 - val_fn: 914746.0000 - val_precision: 0.9641 - val_recall: 0.8452 - val_accuracy: 0.9580 - val_auc: 0.9797
Epoch 80/100
1280/1280 [==============================] - 1953s 2s/step - loss: 0.2669 - tp: 22055420.0000 - fp: 2135651.0000 - tn: 78526352.0000 - fn: 2140199.0000 - precision: 0.9117 - recall: 0.9115 - accuracy: 0.9592 - auc: 0.9815 - val_loss: 0.
2787 - val_tp: 4984180.0000 - val_fp: 197102.0000 - val_tn: 20178072.0000 - val_fn: 855046.0000 - val_precision: 0.9620 - val_recall: 0.8536 - val_accuracy: 0.9599 - val_auc: 0.9800
Epoch 81/100
1280/1280 [==============================] - 1956s 2s/step - loss: 0.2622 - tp: 21705808.0000 - fp: 2077811.0000 - tn: 78966440.0000 - fn: 2107549.0000 - precision: 0.9126 - recall: 0.9115 - accuracy: 0.9601 - auc: 0.9817 - val_loss: 0.
2877 - val_tp: 4988818.0000 - val_fp: 163659.0000 - val_tn: 20154334.0000 - val_fn: 907589.0000 - val_precision: 0.9682 - val_recall: 0.8461 - val_accuracy: 0.9591 - val_auc: 0.9802
Epoch 82/100
1280/1280 [==============================] - 1945s 2s/step - loss: 0.2660 - tp: 22038244.0000 - fp: 2055404.0000 - tn: 78596896.0000 - fn: 2167084.0000 - precision: 0.9147 - recall: 0.9105 - accuracy: 0.9597 - auc: 0.9813 - val_loss: 0.
2860 - val_tp: 4915941.0000 - val_fp: 173380.0000 - val_tn: 20253348.0000 - val_fn: 871720.0000 - val_precision: 0.9659 - val_recall: 0.8494 - val_accuracy: 0.9601 - val_auc: 0.9808
Epoch 83/100
1280/1280 [==============================] - 1960s 2s/step - loss: 0.2620 - tp: 21828352.0000 - fp: 2096385.0000 - tn: 78818296.0000 - fn: 2114614.0000 - precision: 0.9124 - recall: 0.9117 - accuracy: 0.9598 - auc: 0.9819 - val_loss: 0.
2882 - val_tp: 5181476.0000 - val_fp: 172365.0000 - val_tn: 19932176.0000 - val_fn: 928376.0000 - val_precision: 0.9678 - val_recall: 0.8481 - val_accuracy: 0.9580 - val_auc: 0.9790
Epoch 84/100
1280/1280 [==============================] - 1956s 2s/step - loss: 0.2650 - tp: 21972974.0000 - fp: 2043197.0000 - tn: 78688096.0000 - fn: 2153339.0000 - precision: 0.9149 - recall: 0.9107 - accuracy: 0.9600 - auc: 0.9815 - val_loss: 0.
2884 - val_tp: 5125310.0000 - val_fp: 199354.0000 - val_tn: 20006488.0000 - val_fn: 883247.0000 - val_precision: 0.9626 - val_recall: 0.8530 - val_accuracy: 0.9587 - val_auc: 0.9792
Epoch 85/100
1280/1280 [==============================] - 1932s 2s/step - loss: 0.2665 - tp: 21841062.0000 - fp: 2087044.0000 - tn: 78779464.0000 - fn: 2150003.0000 - precision: 0.9128 - recall: 0.9104 - accuracy: 0.9596 - auc: 0.9813 - val_loss: 0.
2990 - val_tp: 4966995.0000 - val_fp: 155834.0000 - val_tn: 20153514.0000 - val_fn: 938068.0000 - val_precision: 0.9696 - val_recall: 0.8411 - val_accuracy: 0.9583 - val_auc: 0.9797
Epoch 86/100
1280/1280 [==============================] - 1950s 2s/step - loss: 0.2613 - tp: 21989296.0000 - fp: 2028370.0000 - tn: 78716320.0000 - fn: 2123700.0000 - precision: 0.9155 - recall: 0.9119 - accuracy: 0.9604 - auc: 0.9819 - val_loss: 0.
3127 - val_tp: 5106508.0000 - val_fp: 142899.0000 - val_tn: 19982462.0000 - val_fn: 982530.0000 - val_precision: 0.9728 - val_recall: 0.8386 - val_accuracy: 0.9571 - val_auc: 0.9803
Epoch 87/100
1280/1280 [==============================] - 1949s 2s/step - loss: 0.2652 - tp: 21800292.0000 - fp: 2065815.0000 - tn: 78849960.0000 - fn: 2141540.0000 - precision: 0.9134 - recall: 0.9106 - accuracy: 0.9599 - auc: 0.9813 - val_loss: 0.
2862 - val_tp: 5040894.0000 - val_fp: 182758.0000 - val_tn: 20103008.0000 - val_fn: 887735.0000 - val_precision: 0.9650 - val_recall: 0.8503 - val_accuracy: 0.9592 - val_auc: 0.9799
Epoch 88/100
1280/1280 [==============================] - 1920s 2s/step - loss: 0.2629 - tp: 21758816.0000 - fp: 2035708.0000 - tn: 78916872.0000 - fn: 2146157.0000 - precision: 0.9144 - recall: 0.9102 - accuracy: 0.9601 - auc: 0.9815 - val_loss: 0.
2883 - val_tp: 5067502.0000 - val_fp: 156913.0000 - val_tn: 20054418.0000 - val_fn: 935565.0000 - val_precision: 0.9700 - val_recall: 0.8442 - val_accuracy: 0.9583 - val_auc: 0.9804
Epoch 89/100
1280/1280 [==============================] - 1902s 1s/step - loss: 0.2661 - tp: 21907896.0000 - fp: 2077818.0000 - tn: 78708656.0000 - fn: 2163245.0000 - precision: 0.9134 - recall: 0.9101 - accuracy: 0.9596 - auc: 0.9814 - val_loss: 0.
3095 - val_tp: 5120703.0000 - val_fp: 137657.0000 - val_tn: 19965744.0000 - val_fn: 990302.0000 - val_precision: 0.9738 - val_recall: 0.8379 - val_accuracy: 0.9570 - val_auc: 0.9806
Epoch 90/100
1280/1280 [==============================] - 1920s 1s/step - loss: 0.2652 - tp: 21984496.0000 - fp: 2114012.0000 - tn: 78621976.0000 - fn: 2137152.0000 - precision: 0.9123 - recall: 0.9114 - accuracy: 0.9595 - auc: 0.9815 - val_loss: 0.
2829 - val_tp: 5208131.0000 - val_fp: 193268.0000 - val_tn: 19946136.0000 - val_fn: 866871.0000 - val_precision: 0.9642 - val_recall: 0.8573 - val_accuracy: 0.9596 - val_auc: 0.9809
Epoch 91/100
1280/1280 [==============================] - 1917s 1s/step - loss: 0.2631 - tp: 21814944.0000 - fp: 2037460.0000 - tn: 78857032.0000 - fn: 2148231.0000 - precision: 0.9146 - recall: 0.9104 - accuracy: 0.9601 - auc: 0.9817 - val_loss: 0.
2921 - val_tp: 4988873.0000 - val_fp: 154974.0000 - val_tn: 20127408.0000 - val_fn: 943141.0000 - val_precision: 0.9699 - val_recall: 0.8410 - val_accuracy: 0.9581 - val_auc: 0.9797
Epoch 92/100
1280/1280 [==============================] - 1920s 1s/step - loss: 0.2641 - tp: 21926682.0000 - fp: 2066839.0000 - tn: 78722320.0000 - fn: 2141742.0000 - precision: 0.9139 - recall: 0.9110 - accuracy: 0.9599 - auc: 0.9816 - val_loss: 0.
3016 - val_tp: 5074157.0000 - val_fp: 161845.0000 - val_tn: 20024836.0000 - val_fn: 953570.0000 - val_precision: 0.9691 - val_recall: 0.8418 - val_accuracy: 0.9575 - val_auc: 0.9794
Epoch 93/100
1280/1280 [==============================] - 1915s 1s/step - loss: 0.2639 - tp: 21858124.0000 - fp: 2086217.0000 - tn: 78785032.0000 - fn: 2128289.0000 - precision: 0.9129 - recall: 0.9113 - accuracy: 0.9598 - auc: 0.9817 - val_loss: 0.
3349 - val_tp: 5110189.0000 - val_fp: 138626.0000 - val_tn: 19910190.0000 - val_fn: 1055394.0000 - val_precision: 0.9736 - val_recall: 0.8288 - val_accuracy: 0.9545 - val_auc: 0.9781
Epoch 94/100
1280/1280 [==============================] - 1932s 2s/step - loss: 0.2661 - tp: 21972080.0000 - fp: 2067147.0000 - tn: 78645520.0000 - fn: 2172891.0000 - precision: 0.9140 - recall: 0.9100 - accuracy: 0.9596 - auc: 0.9814 - val_loss: 0.
3010 - val_tp: 5379887.0000 - val_fp: 190356.0000 - val_tn: 19720614.0000 - val_fn: 923543.0000 - val_precision: 0.9658 - val_recall: 0.8535 - val_accuracy: 0.9575 - val_auc: 0.9800
Epoch 95/100
1280/1280 [==============================] - 1925s 2s/step - loss: 0.2641 - tp: 21773760.0000 - fp: 2069301.0000 - tn: 78892664.0000 - fn: 2121866.0000 - precision: 0.9132 - recall: 0.9112 - accuracy: 0.9600 - auc: 0.9814 - val_loss: 0.
2834 - val_tp: 5148394.0000 - val_fp: 196890.0000 - val_tn: 20004664.0000 - val_fn: 864450.0000 - val_precision: 0.9632 - val_recall: 0.8562 - val_accuracy: 0.9595 - val_auc: 0.9803
Epoch 96/100
1280/1280 [==============================] - 1924s 2s/step - loss: 0.2620 - tp: 21883636.0000 - fp: 2083221.0000 - tn: 78785256.0000 - fn: 2105458.0000 - precision: 0.9131 - recall: 0.9122 - accuracy: 0.9601 - auc: 0.9819 - val_loss: 0.
3238 - val_tp: 5063114.0000 - val_fp: 150917.0000 - val_tn: 20012972.0000 - val_fn: 987390.0000 - val_precision: 0.9711 - val_recall: 0.8368 - val_accuracy: 0.9566 - val_auc: 0.9792
Epoch 97/100
1280/1280 [==============================] - 1926s 2s/step - loss: 0.2629 - tp: 21759104.0000 - fp: 2051085.0000 - tn: 78912920.0000 - fn: 2134447.0000 - precision: 0.9139 - recall: 0.9107 - accuracy: 0.9601 - auc: 0.9815 - val_loss: 0.
2802 - val_tp: 5228317.0000 - val_fp: 190075.0000 - val_tn: 19926608.0000 - val_fn: 869396.0000 - val_precision: 0.9649 - val_recall: 0.8574 - val_accuracy: 0.9596 - val_auc: 0.9809
Epoch 98/100
1280/1280 [==============================] - 1929s 2s/step - loss: 0.2630 - tp: 21930218.0000 - fp: 2067388.0000 - tn: 78741992.0000 - fn: 2117977.0000 - precision: 0.9139 - recall: 0.9119 - accuracy: 0.9601 - auc: 0.9817 - val_loss: 0.
3097 - val_tp: 4899275.0000 - val_fp: 135668.0000 - val_tn: 20163436.0000 - val_fn: 1016018.0000 - val_precision: 0.9731 - val_recall: 0.8282 - val_accuracy: 0.9561 - val_auc: 0.9789
Epoch 99/100
1280/1280 [==============================] - 1957s 2s/step - loss: 0.2648 - tp: 21993828.0000 - fp: 2052826.0000 - tn: 78663616.0000 - fn: 2147339.0000 - precision: 0.9146 - recall: 0.9111 - accuracy: 0.9599 - auc: 0.9816 - val_loss: 0.
2889 - val_tp: 5106607.0000 - val_fp: 161956.0000 - val_tn: 20025448.0000 - val_fn: 920399.0000 - val_precision: 0.9693 - val_recall: 0.8473 - val_accuracy: 0.9587 - val_auc: 0.9814
Epoch 100/100
1280/1280 [==============================] - 1941s 2s/step - loss: 0.2598 - tp: 21924942.0000 - fp: 2029354.0000 - tn: 78777592.0000 - fn: 2125714.0000 - precision: 0.9153 - recall: 0.9116 - accuracy: 0.9604 - auc: 0.9819 - val_loss: 0.
3001 - val_tp: 5185959.0000 - val_fp: 162584.0000 - val_tn: 19884838.0000 - val_fn: 981025.0000 - val_precision: 0.9696 - val_recall: 0.8409 - val_accuracy: 0.9564 - val_auc: 0.9800
400/400 [==============================] - 121s 302ms/step - loss: 0.3130 - tp: 6560497.0000 - fp: 267592.0000 - tn: 24822684.0000 - fn: 1117239.0000 - precision: 0.9608 - recall: 0.8545 - accuracy: 0.9577 - auc: 0.9801
2020/06/14 20:28:16 INFO mlflow.projects: === Run (ID '037e1d9e4ad74784974f4aaac11138cc') succeeded ===

2.1.11 read out logs experiment 2

2.1.11.1 read out mlflow logs using CLI
conda activate tensorflow_env
cd Programme/drmed-git
export MLFLOW_EXPERIMENT_NAME=exp-310520-unet
export MLFLOW_TRACKING_URI=file:./data/mlruns
mlflow experiments list
  Experiment Id  Name             Artifact Location
---------------  ---------------  --------------------
              0  exp-310520-unet  file:./data/mlruns/0
              1  exp-devtest      file:./data/mlruns/1
mlflow runs list --experiment-id 0
Date                      Name    ID
------------------------  ------  --------------------------------
2020-06-12 14:20:30 CEST          037e1d9e4ad74784974f4aaac11138cc
2020-05-31 21:39:51 CEST          1aefda1366f04f5da5d1fc2241ad9208
export EXP2=037e1d9e4ad74784974f4aaac11138cc

I have accidentally run the model with the wrong experiment ID. I'll check if I can change the experiment. I will do it later with mlflow.tracking.MlflowClient().rename_experiment()

mlflow artifacts list -r $EXP2
mlflow artifacts list -r $EXP2 -a model
mlflow artifacts list -r $EXP2 -a model_summary.txt
mlflow artifacts list -r $EXP2 -a tensorboard_logs/train/
(tensorflow_env) [lex@Topialex drmed-git]$ mlflow artifacts list -r $EXP2
[{
  "path": "model",
  "is_dir": true
}, {
  "path": "model_summary.txt",
  "is_dir": false,
  "file_size": "10895"
}, {
  "path": "tensorboard_logs",
  "is_dir": true
}]

(tensorflow_env) [lex@Topialex drmed-git]$ mlflow artifacts list -r $EXP2 -a model
[{
  "path": "model/MLmodel",
  "is_dir": false,
  "file_size": "317"
}, {
  "path": "model/conda.yaml",
  "is_dir": false,
  "file_size": "125"
}, {
  "path": "model/data",
  "is_dir": true
}]
(tensorflow_env) [lex@Topialex drmed-git]$ mlflow artifacts list -r $EXP2 -a model_summary.txt
[]

(tensorflow_env) [lex@Topialex drmed-git]$ mlflow artifacts list -r $EXP2 -a tensorboard_logs/train/
[{
  "path": "tensorboard_logs/train/events.out.tfevents.1591964813.node151.309868.9202.v2",
  "is_dir": false,
  "file_size": "20270485"
}, {
  "path": "tensorboard_logs/train/events.out.tfevents.1591964827.node151.profile-empty",
  "is_dir": false,
  "file_size": "40"
}, {
  "path": "tensorboard_logs/train/plugins",
  "is_dir": true
}]
mlflow artifacts download -r $EXP2
mlflow artifacts download -r $EXP2 -a model
mlflow artifacts download -r $EXP2 -a model_summary.txt
mlflow artifacts download -r $EXP2 -a tensorboard_logs
/home/lex/Programme/drmed-git/data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts
/home/lex/Programme/drmed-git/data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts/model
/home/lex/Programme/drmed-git/data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts/model_summary.txt
/home/lex/Programme/drmed-git/data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts/tensorboard_logs
mlflow runs describe --run-id $EXP2
{
    "info": {
        "artifact_uri": "file:./data/mlruns/0/037e1d9e4ad74784974f4aaac11138cc/artifacts",
        "end_time": 1592159296429,
        "experiment_id": "0",
        "lifecycle_stage": "active",
        "run_id": "037e1d9e4ad74784974f4aaac11138cc",
        "run_uuid": "037e1d9e4ad74784974f4aaac11138cc",
        "start_time": 1591964430233,
        "status": "FINISHED",
        "user_id": "ye53nis"
    },
    "data": {
        "metrics": {
            "learning rate": 1e-14,
            "val_loss": 0.3001052141189575,
            "precision": 0.9152822494506836,
            "fp": 2029354.0,
            "loss": 0.2597578763961792,
            "val_tp": 5185959.0,
            "recall": 0.9116151332855225,
            "accuracy": 0.9603750705718994,
            "val_precision": 0.9696021676063538,
            "val_recall": 0.8409230709075928,
            "lr": 1e-14,
            "fn": 2125714.0,
            "auc": 0.9819398522377014,
            "val_fp": 162584.0,
            "val_accuracy": 0.9563749432563782,
            "val_tn": 19884838.0,
            "val_auc": 0.9799740314483643,
            "val_fn": 981025.0,
            "tn": 78777592.0,
            "tp": 21924942.0
        },
        "params": {
            "opt_beta_1": "0.9",
            "validation_steps": "320",
            "opt_learning_rate": "0.001",
            "fluotracify_path": "/beegfs/ye53nis/drmed-git/src/",
            "opt_amsgrad": "False",
            "frac_val": "0.2",
            "batch_size": "5",
            "epochs": "100",
            "opt_decay": "0.0",
            "steps_per_epoch": "1280",
            "length_delimiter": "16384",
            "opt_name": "Adam",
            "opt_beta_2": "0.999",
            "csv_path": "/beegfs/ye53nis/saves/firstartefact_Sep2019",
            "learning_rate": "None",
            "opt_epsilon": "1e-07"
        },
        "tags": {
            "mlflow.source.git.repoURL": "https://github.com/aseltmann/fluotracify",
            "mlflow.source.type": "PROJECT",
            "mlflow.source.name": "file:///beegfs/ye53nis/drmed-git",
            "mlflow.user": "ye53nis",
            "mlflow.source.git.commit": "09c1d2a1bc083695026b46c74ea175c9168bb5f2",
            "mlflow.gitRepoURL": "https://github.com/aseltmann/fluotracify",
            "mlflow.project.backend": "local",
            "mlflow.project.env": "conda",
            "mlflow.project.entryPoint": "main",
            "mlflow.log-model.history": "[{
                \"run_id\": \"037e1d9e4ad74784974f4aaac11138cc\",
                \"artifact_path\": \"model\",
                \"utc_time_created\": \"2020-06-14 18:26:02.386982\",
                \"flavors\": {
                    \"keras\": {
                        \"keras_module\": \"tensorflow.keras\",
                        \"keras_version\": \"2.2.4-tf\",
                        \"data\": \"data\"
                    },
                    \"python_function\": {
                        \"loader_module\": \"mlflow.keras\",
                        \"python_version\": \"3.8.3\",
                        \"data\": \"data\",
                        \"env\": \"conda.yaml\"
                    }
                }
            }]"
        }
    }
}
tensorboard --logdir=data/mlruns/0/$EXP2/artifacts/tensorboard_logs
mlflow ui --backend-store-uri file:///home/lex/Programme/drmed-git/data/mlruns
2.1.11.2 plots via mlflow ui, comparing run 1 and 2

AUC: run2_auc.png Loss: run2_loss.png Precision / Recall: run2_prec_recall.png

2.1.11.3 plots via tensorboard ui

Prediction 03 prediction_03.png Prediction 20 prediction_20.png Prediction 24 prediction_24.png Prediction 32 prediction_32.png Prediction 99 prediction_99.png Distribution of Conv1D-Kernel (final layer) distribution_conv1d-kernel_finallayer.png Histograms of Conv1D-Kernel (final layer) histograms_conv1d-kernel_finallayer.png

2.1.11.4 for next runs
  • more steps = good: better recall
  • strong fluctuation of validation metrics in beginning and already convergence after ~30 Epochs for the second run → maybe use learning rate a bit more gently and start the lr schedule with lower rates
  • judging from the tensorboard histograms, the model architecture can clearly be made simpler
  • doing some background reading: in this SO thread it is argued that while using Adam, an extensive learning rate schedule, as I used, should not really be necessary.
    • Adam (adaptive moment estimation) updates any parameter with an individual learning rate
    • every learning rate can vary from 0 (no update) to the given learning rate as an upper limit
    • others argue, that it definitely helps (or that you have to start with a very low lr, which I can confirm from own experience) and that it especially could help reducing the loss in the late steps of training
    • finding optimal lr:
      • start with very very low lr, than increase till loss stops decreasing, and look at where the slope of the loss curve and pick the learning rate that is associated with the fastest decrease
  1. effect of batch size
    • see effect of batch size
    • from pracitcal point: larger batch size → computational speedups from parallelism of GPUs. In theory, using a batch equal to the entire dataset guarantees convergence to global optima of objective function - hower on cost of slower, empirical convergence
    • well known: too large of a batch size → poor generalization (hoffer, hubara, soudry argue here that this is not inherently true).
      • why poor gen: competing gradients of different training examples → sequential optimization is easier than simultaneous optimization in complex, high dimensional parameter spaces
    • smaller batch size pro: faster convergence to "good" solutions
    • smaller batch size con: model not guaranteed to converge to global optima, will bounce around, staying outside some \(\epsilon\) - ball of the optima, where \(\epsilon\) depends on the ratio of the batch size to the data size
    • if no computational constraints: start at small batch size, steadily grow batch size through training.
    • non-convex models: "sweet spot" between batch size of 1 (bad, "noisy", nn prone to overfitting) and entire training dataset.
    • on other hand: "noise" in small batch size might be good → "tug-and-pull" dynamic which might prevent nn from overfitting
    • what is "large number of epochs" → number of epochs such that any further training provides little to no boost in test accuracy → difficult to determine → best guess
    • example MNIST: bs=64 gets test acc of 98%, bs=1024 gets test acc of 95% → increasing learning rate can compensate for larger batch sizes (for bs=1024 from 0.01 to 0.1 → problem solved) → could be more difficult in more complex datasets
    • example MNIST: starting with large batch size does not "get the model stuck" in bad local optimums, better test accuracy can be achieved anytime by switching to lower batch size or higher learning rate
    • investigation of how stuff is updated with large batch size:
      • larger batch sizes make larger gradient steps than smaller batch sizes for the same number of samples seen
      • for same average euclidean norm distance from initial weights of the model, larger batch sizes have larger variance in the distance → if smaller batch sizes are more "noisy", this would be the other way around
      • conclusion:
        • large batch size → model makes very large gradient updates and very small gradient updates → size depends on which particular samples are drawn.
        • Small batch size → model makes updates of about the same size → weakly dependend on which particular samples are drawn.
    • better solutions can be far away from inital weights → if loss is averaged over the batch then large batch sizes simply do not allow the model to travel far enough to reach the better solutions for the same number of training epochs → you take fewer steps → increasing lr makes steps larger → and then they can even move further after seeing the same number of samples
    • Adam vs SGD
      • Adam claims: insensitivity to weight initialization and initial learning rate choice
      • finding: adam finds solutions with much larger weights, which might explain why it has lower test accuracy and is not generalizing as well → this is why weight decay is recommended with Adam
      • in SGD the weights are initialized to approx the magnitude you want them to be and most of the learning is shuffling the weights along the hyper-sphere of the initial radius
      • in Adam, the model ignores the initialization
    • hoffer et al
      • propose that in SGD, initial learning phase can be described using high-dimensional "random walk on a random potential" process, with an "ultra-slow" logarithmic increase in the distance of the weights from their initialization (observed empirically)
      • remedies
        1. Use SGD with momentum, gradient clipping, and a decreasing learning rate schedule
        2. adapt learning rate with batch size (e.g. square root scaling)
        3. compute batch-norm statistics over several partitions ("ghost batch-norm")
        4. use sufficient number of high learning rate training iterations
      • the "generalization gap" problem is not related to the batch size but rather the amount of updates
    • Smith et al, Google Brain
      • instead of learning rate decay, increase batch size with training → works with Adam, Nesterov momentum, SGD with momentum, SGD → reaches equivalent test accuracies after same no of epochs, but with fewer parameter updates → greater parallelism (good for GPU), shorter training time
  2. using dropout layers with Unet
    • Bartolome et al did it in DeepCell for automatic nuclei detection
      • encoding block:
        • maxpool
        • 2 times conv+ELU+batchnorm
      • decoding block:
        • upsample
        • dropout (0.3)
        • 2 times conv+ELU
      • last block
        • 1x1 conv
        • sigmoid
    • from this reddit
      • Srivastava/Hinton dropout paper: additional gain in performacne obtained by adding dropout in conv layers is worth noting (3% to 2.55%). Dropout in lower layers help, because it provides noisy inputs for the higher fully connected layers which prevents them from overfitting → they use 0.7 prob for conv dropout and 0.5 for FC

2.1.12 use model from run 2 to correct data

2.1.12.1 simulated data from the test set

brightbursts_correction_by_unet_histogram_050_tt_200702.svg

2.1.12.2 experimental data from Pablo (structured experiment)

ptu_brightbursts_correction_by_unet_histogram_010_tt_200704_400traces.svg

  1. biological metadata
    • Who took the data: Pablo Carravilla in November 2019 in Oxford
    • a bit of 400 "dirty" and 400 "clean" curves
    • AF488: small dye, homogeneous signal
    • "clean"
      • in folders GroupMeas_5" and GroupMeas_6
      • Hs-PEX5-eGFP - PEX5 from Homo sapiens, labelled with eGFP
      • typical filename: 20 nM AF488147_T1754s_1.ptu
    • "dirty"
      • in folders GroupMeas_1 to GroupMeas_4_4
      • Tp-PEX5-eGFP - PEX5 from Trypanosoma brucei, labelled with eGFP
      • typical filename: DiO LUV 10uM in 20 nM AF48816_T197s_1.ptu
      • Dio LUV: big vesicles, spikes
  2. TODO microscope metadata - paste from pickled pandas dfs
  3. anecdotal, semi-structured plotting of correlation curves
    • first: "clean" data of Hs-PEX5-eGFP
    • from plotting 10 traces with the respective binning windows using correct_correlation_by_unet_prediction:
      • prediction didn't find anything - awesome! So no values were removed and correlations are "pure"
      • for correlation at \(1ms\) binning window:
        • transit time ~\(0.25...0.5ms\)
        • diffusion coefficient ~\(22.5...45 \mu m^2 / s\)
    • run2_hs-pex5-egfp_1mscorr_1.png
      • for correlation at \(100\mu s\) binning window:
        • transit time ~\(0.4...0.53ms\)
        • diffusion coefficient ~\(21...29 \mu m^2 / s\)
        • This is probably the most accurate!
    • run2_hs-pex5-egfp_100uscorr_1.png
      • for correlation at \(10\mu s\) and at 1us binning window:
        • transit time ~\(0.09ms\)
        • diffusion coefficient ~\(120...133\mu m^2 / s\)
        • fit didn't look too good - probably something is not working correctly here with multipletau. Looks like the start of the correlation curve is flattening, but multipletau fits an increase
    • run2_hs-pex5-egfp_10uscorr_1.png
      • second: "dirty" data of Tb-PEX5-eGFP
      • from plotting 10 traces with the respective binning windows using correct_correlation_by_unet_prediction:
        • for correlation at \(1ms\) binning wnidow:
          • transit time
            • without correction: ~\(6...20ms\)
            • with correction: ~\(2...5ms\)
          • diffusion coefficients:
            • without correction: ~\(0.6...1.9\mu m^2 / s\)
            • with correction: ~\(2...6.3 \mu m^2 / s\)
    • run2_tb-pex5-egfp_1mscorr_2.png
      • for correlation at \(100 \mu s\) binninw window:
        • transit time
          • without correction: ~\(43...149ms\)
          • with correction: ~\(1.6...42ms\)
        • diffusion coefficients:
          • without correction: ~\(0.08...0.26\mu m^2/s\)
          • with correction: ~\(0.3...7\mu m^2 / s\)
    • run2_tb-pex5-egfp_100uscorr_1.png
      • for correlation at \(10\mu s\) binning window:
        • fitting looks HORRIBLE for the traces corrected by prediction, while looking "okayish" for traces without correction - even though the values are bad.
        • transit time
          • without correction: ~\(153...1329ms\)
          • with correction: ~\(0.2...346ms\)
        • diffusion coefficients:
          • without correction: ~\(0.008...0.07 \mu m^2 / s\)
          • with correction: ~\(0.03...52\mu m^2 / s\)
    • run2_tb-pex5-egfp_10uscorr_3.png
      • for correlation at \(1\mu s\) binning window (only 3 traces, because ):
        • fits have to be better!
    • run2_tb-pex5-egfp_1uscorr_1.png

2.1.13 git exp 3

git status
git log -1
(base) [ye53nis@node161 drmed-git]$ git status
# On branch exp-310520-unet
# Untracked files:
# ...
nothing added to commit but untracked files present (use "git add" to track)

(base) [ye53nis@node161 drmed-git]$ git log -1
commit 193d9e3f8d126828b253121e613ddaf5363d7c3d
Author: Apoplex <oligolex@vivaldi.net>
Date:   Wed Jun 17 12:09:44 2020 +0200

    Move experiment to correct mlflow id

2.1.14 experimental run 3 - bs=3

conda activate tensorflow_nightly
cd /beegfs/ye53nis/drmed-git
export MLFLOW_EXPERIMENT_NAME=exp-310520-unet
export MLFLOW_TRACKING_URI=file:./data/mlruns
mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src -P batch_size=3 -P epochs=40 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact_Sep2019 -P steps_per_epoch=2100 -P validation_steps=320
2020/06/20 00:43:51 INFO mlflow.projects: === Created directory /tmp/tmpxwtlwi2b for downloading remote URIs passed to arguments of type 'path' ===
2020/06/20 00:43:51 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/fluotracify/training/train.py /
beegfs/ye53nis/drmed-git/src 3 0.2 16384 None 40 /beegfs/ye53nis/saves/firstartefact_Sep2019 2100 320' in run with ID '1e98b3ed2e1d421da9592058bd5587a8' ===
2020-06-20 00:44:08.580506: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-06-20 00:44:08.580645: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favour of importlib; see the modul
e's documentation for alternative uses
  import imp
2.3.0-dev20200527
2020-06-20 00:44:36.776071: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-06-20 00:44:36.776132: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-06-20 00:44:36.776174: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node161): /proc/driver/nvidia/version does not exist
GPUs:  []
train 0 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set027.csv
train 1 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set087.csv
train 2 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set003.csv
train 3 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set056.csv
train 4 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set076.csv
train 5 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set094.csv
train 6 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set017.csv
train 7 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set074.csv
train 8 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set055.csv
train 9 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set096.csv
train 10 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set054.csv
train 11 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set093.csv
train 12 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set079.csv
train 13 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set014.csv
train 14 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set008.csv
train 15 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set031.csv
train 16 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set023.csv
train 17 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set025.csv
train 18 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set034.csv
train 19 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set009.csv
train 20 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set044.csv
train 21 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set063.csv
train 22 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set004.csv
train 23 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set072.csv
train 24 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set046.csv
train 25 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set049.csv
train 26 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set007.csv
train 27 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set100.csv
train 28 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set083.csv
train 29 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set077.csv
train 30 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set061.csv
train 31 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set081.csv
train 32 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set091.csv
train 33 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set069.csv
train 34 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set052.csv
train 35 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set028.csv
train 36 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set019.csv
train 37 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set057.csv
train 38 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set064.csv
train 39 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set075.csv
train 40 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set002.csv
train 41 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set062.csv
train 42 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set043.csv
train 43 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set042.csv
train 44 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set005.csv
train 45 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set016.csv
train 46 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set018.csv
train 47 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set041.csv
train 48 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set039.csv
train 49 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set006.csv
train 50 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set092.csv
train 51 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set060.csv
train 52 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set001.csv
train 53 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set035.csv
train 54 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set029.csv
train 55 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set051.csv
train 56 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set012.csv
train 57 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set036.csv
train 58 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set024.csv
train 59 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set053.csv
train 60 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set011.csv
train 61 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set032.csv
train 62 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set067.csv
train 63 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set058.csv
train 64 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set080.csv
train 65 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set086.csv
train 66 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set033.csv
train 67 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set085.csv
train 68 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set015.csv
train 69 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set090.csv
train 70 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set020.csv
train 71 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set030.csv
train 72 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set050.csv
train 73 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set098.csv
train 74 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set099.csv
train 75 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set070.csv
train 76 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set021.csv
train 77 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set095.csv
train 78 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set073.csv
train 79 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set078.csv
test 80 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set026.csv
test 81 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set038.csv
test 82 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set082.csv
test 83 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set047.csv
test 84 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set040.csv
test 85 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set066.csv
test 86 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set059.csv
test 87 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set013.csv
test 88 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set089.csv
test 89 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set071.csv
test 90 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set088.csv
test 91 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set037.csv
test 92 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set022.csv
test 93 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set084.csv
test 94 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set010.csv
test 95 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set097.csv
test 96 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set068.csv
test 97 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set065.csv
test 98 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set048.csv
test 99 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set045.csv
shapes of feature dataframe: (20000, 8000) and label dataframe: (20000, 8000)
shapes of feature dataframe: (20000, 2000) and label dataframe: (20000, 2000)

for each 20,000 timestap trace there are the following numbers of corrupted timesteps:
 label001_1    6286
label001_1    2568
label001_1    4495
label001_1    4414
label001_1    1105
dtype: int64
2020-06-20 00:49:48.824441: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-06-20 00:49:48.834205: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz
2020-06-20 00:49:48.835707: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5643aca77c10 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-20 00:49:48.835736: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
number of training examples: 6400, number of validation examples: 1600

------------------------
number of test examples: 2000

input - shape:   (None, 16384, 1)
output - shape:  (None, 16384, 1)
2020-06-20 00:49:53.167773: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the ABCs from 'collections' instead
 of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  if not isinstance(wrapped_dict, collections.Mapping):
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature()
 or inspect.getfullargspec()
  all_param_names, _, _, all_default_values = inspect.getargspec(fn)  # pylint: disable=W1505
Epoch 1/40
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from
'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  if not isinstance(values, collections.Sequence):
   1/2100 [..............................] - ETA: 0s - loss: 1.4336 - tp: 11096.0000 - fp: 13747.0000 - tn: 16434.0000 - fn: 7875.0000 - precision: 0.4466 - recall: 0.5849 - accuracy: 0.5601 - auc: 0.59502020-06-20 00:50:06.341154: I te
nsorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager.profiler) is deprecated and w
ill be removed after 2020-07-01.
Instructions for updating:
use `tf.profiler.experimental.stop` instead.
2020-06-20 00:50:07.661012: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_06_20_00_50_07
2020-06-20 00:50:07.677958: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_06_20_00_50_07/node161.trace.json.gz
2020-06-20 00:50:07.711562: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_06_20_00_50_07
2020-06-20 00:50:07.711709: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_06_20_00_50_07/node161.memory_profile.json.gz
2020-06-20 00:50:07.713980: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_06_20_00_50_07Dumped tool data for xplane.pb to /tmp/tb/train/plugins/profile/2020_06_20_0
0_50_07/node161.xplane.pb
Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_06_20_00_50_07/node161.overview_page.pb
Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_06_20_00_50_07/node161.input_pipeline.pb
Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_06_20_00_50_07/node161.tensorflow_stats.pb
Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_06_20_00_50_07/node161.kernel_stats.pb

2100/2100 [==============================] - 2619s 1s/step - loss: 1.1651 - tp: 14036256.0000 - fp: 5361485.0000 - tn: 74321616.0000 - fn: 9499881.0000 - precision: 0.7236 - recall: 0.5964 - accuracy: 0.8560 - auc: 0.8712 - val_loss: 6.
7480 - val_tp: 3658812.0000 - val_fp: 7608396.0000 - val_tn: 4305440.0000 - val_fn: 155992.0000 - val_precision: 0.3247 - val_recall: 0.9591 - val_accuracy: 0.5064 - val_auc: 0.7199
Epoch 2/40
2100/2100 [==============================] - 2616s 1s/step - loss: 0.7522 - tp: 16978660.0000 - fp: 4731235.0000 - tn: 74935072.0000 - fn: 6574314.0000 - precision: 0.7821 - recall: 0.7209 - accuracy: 0.8905 - auc: 0.9195 - val_loss: 2.
3699 - val_tp: 1583199.0000 - val_fp: 169213.0000 - val_tn: 11937019.0000 - val_fn: 2039209.0000 - val_precision: 0.9034 - val_recall: 0.4371 - val_accuracy: 0.8596 - val_auc: 0.7654
Epoch 3/40
2100/2100 [==============================] - 2615s 1s/step - loss: 0.5253 - tp: 19558076.0000 - fp: 3583985.0000 - tn: 75779848.0000 - fn: 4297304.0000 - precision: 0.8451 - recall: 0.8199 - accuracy: 0.9236 - auc: 0.9546 - val_loss: 20
.3139 - val_tp: 3551544.0000 - val_fp: 6920318.0000 - val_tn: 5181055.0000 - val_fn: 75723.0000 - val_precision: 0.3392 - val_recall: 0.9791 - val_accuracy: 0.5552 - val_auc: 0.7820
Epoch 4/40
2100/2100 [==============================] - 2593s 1s/step - loss: 0.4607 - tp: 19722116.0000 - fp: 3070380.0000 - tn: 76557328.0000 - fn: 3869410.0000 - precision: 0.8653 - recall: 0.8360 - accuracy: 0.9328 - auc: 0.9621 - val_loss: 0.
6783 - val_tp: 2809999.0000 - val_fp: 1055013.0000 - val_tn: 11273173.0000 - val_fn: 590455.0000 - val_precision: 0.7270 - val_recall: 0.8264 - val_accuracy: 0.8954 - val_auc: 0.9256
Epoch 5/40
2100/2100 [==============================] - 2579s 1s/step - loss: 0.4347 - tp: 20035524.0000 - fp: 2906267.0000 - tn: 76615392.0000 - fn: 3662016.0000 - precision: 0.8733 - recall: 0.8455 - accuracy: 0.9364 - auc: 0.9644 - val_loss: 0.
9250 - val_tp: 2881407.0000 - val_fp: 1446289.0000 - val_tn: 10683025.0000 - val_fn: 717919.0000 - val_precision: 0.6658 - val_recall: 0.8005 - val_accuracy: 0.8624 - val_auc: 0.9110
Epoch 6/40
2100/2100 [==============================] - 2587s 1s/step - loss: 0.3938 - tp: 20159010.0000 - fp: 2640041.0000 - tn: 77171896.0000 - fn: 3248240.0000 - precision: 0.8842 - recall: 0.8612 - accuracy: 0.9430 - auc: 0.9692 - val_loss: 0.
9069 - val_tp: 3350060.0000 - val_fp: 2334741.0000 - val_tn: 9777228.0000 - val_fn: 266611.0000 - val_precision: 0.5893 - val_recall: 0.9263 - val_accuracy: 0.8346 - val_auc: 0.9424
Epoch 7/40
2100/2100 [==============================] - 2596s 1s/step - loss: 0.3898 - tp: 20604918.0000 - fp: 2623827.0000 - tn: 76744272.0000 - fn: 3246144.0000 - precision: 0.8870 - recall: 0.8639 - accuracy: 0.9431 - auc: 0.9700 - val_loss: 1.
1869 - val_tp: 2154011.0000 - val_fp: 161830.0000 - val_tn: 11907682.0000 - val_fn: 1505117.0000 - val_precision: 0.9301 - val_recall: 0.5887 - val_accuracy: 0.8940 - val_auc: 0.8640
Epoch 8/40
2100/2100 [==============================] - 2585s 1s/step - loss: 0.3615 - tp: 20559308.0000 - fp: 2451944.0000 - tn: 77195760.0000 - fn: 3012324.0000 - precision: 0.8934 - recall: 0.8722 - accuracy: 0.9471 - auc: 0.9725 - val_loss: 2.
2166 - val_tp: 457245.0000 - val_fp: 0.0000e+00 - val_tn: 12111846.0000 - val_fn: 3159549.0000 - val_precision: 1.0000 - val_recall: 0.1264 - val_accuracy: 0.7991 - val_auc: 0.7373
Epoch 9/40
2100/2100 [==============================] - 2575s 1s/step - loss: 0.3614 - tp: 20474248.0000 - fp: 2479592.0000 - tn: 77192832.0000 - fn: 3072574.0000 - precision: 0.8920 - recall: 0.8695 - accuracy: 0.9462 - auc: 0.9720 - val_loss: 1.
6778 - val_tp: 3153657.0000 - val_fp: 3799634.0000 - val_tn: 8459831.0000 - val_fn: 315518.0000 - val_precision: 0.4535 - val_recall: 0.9091 - val_accuracy: 0.7384 - val_auc: 0.8806
Epoch 10/40
2100/2100 [==============================] - 2574s 1s/step - loss: 0.3525 - tp: 20915356.0000 - fp: 2417250.0000 - tn: 76968064.0000 - fn: 2918540.0000 - precision: 0.8964 - recall: 0.8775 - accuracy: 0.9483 - auc: 0.9734 - val_loss: 0.
7001 - val_tp: 3011358.0000 - val_fp: 1423566.0000 - val_tn: 10810936.0000 - val_fn: 482780.0000 - val_precision: 0.6790 - val_recall: 0.8618 - val_accuracy: 0.8788 - val_auc: 0.9191
Epoch 11/40
2100/2100 [==============================] - 2583s 1s/step - loss: 0.3536 - tp: 20693556.0000 - fp: 2409057.0000 - tn: 77110336.0000 - fn: 3006188.0000 - precision: 0.8957 - recall: 0.8732 - accuracy: 0.9475 - auc: 0.9731 - val_loss: 0.
8146 - val_tp: 3341267.0000 - val_fp: 1466467.0000 - val_tn: 10619280.0000 - val_fn: 301626.0000 - val_precision: 0.6950 - val_recall: 0.9172 - val_accuracy: 0.8876 - val_auc: 0.9452
Epoch 12/40
2100/2100 [==============================] - 2559s 1s/step - loss: 0.3165 - tp: 21370760.0000 - fp: 2314648.0000 - tn: 77018928.0000 - fn: 2514801.0000 - precision: 0.9023 - recall: 0.8947 - accuracy: 0.9532 - auc: 0.9771 - val_loss: 0.
3977 - val_tp: 2929496.0000 - val_fp: 337784.0000 - val_tn: 11799466.0000 - val_fn: 661894.0000 - val_precision: 0.8966 - val_recall: 0.8157 - val_accuracy: 0.9364 - val_auc: 0.9624
Epoch 13/40
2100/2100 [==============================] - 2555s 1s/step - loss: 0.2959 - tp: 21260244.0000 - fp: 2183093.0000 - tn: 77404592.0000 - fn: 2371187.0000 - precision: 0.9069 - recall: 0.8997 - accuracy: 0.9559 - auc: 0.9785 - val_loss: 0.
4174 - val_tp: 2796648.0000 - val_fp: 277583.0000 - val_tn: 11853440.0000 - val_fn: 800969.0000 - val_precision: 0.9097 - val_recall: 0.7774 - val_accuracy: 0.9314 - val_auc: 0.9595
Epoch 14/40
2100/2100 [==============================] - 2567s 1s/step - loss: 0.2983 - tp: 21268994.0000 - fp: 2193443.0000 - tn: 77331256.0000 - fn: 2425591.0000 - precision: 0.9065 - recall: 0.8976 - accuracy: 0.9552 - auc: 0.9784 - val_loss: 0.
3715 - val_tp: 3222237.0000 - val_fp: 299190.0000 - val_tn: 11646772.0000 - val_fn: 560441.0000 - val_precision: 0.9150 - val_recall: 0.8518 - val_accuracy: 0.9453 - val_auc: 0.9688
Epoch 15/40
2100/2100 [==============================] - 2562s 1s/step - loss: 0.2919 - tp: 21573232.0000 - fp: 2138857.0000 - tn: 77160424.0000 - fn: 2346711.0000 - precision: 0.9098 - recall: 0.9019 - accuracy: 0.9565 - auc: 0.9791 - val_loss: 0.
4316 - val_tp: 3028795.0000 - val_fp: 656530.0000 - val_tn: 11568012.0000 - val_fn: 475303.0000 - val_precision: 0.8219 - val_recall: 0.8644 - val_accuracy: 0.9280 - val_auc: 0.9580
Epoch 16/40
2100/2100 [==============================] - 2544s 1s/step - loss: 0.2915 - tp: 21637290.0000 - fp: 2191368.0000 - tn: 77039000.0000 - fn: 2351546.0000 - precision: 0.9080 - recall: 0.9020 - accuracy: 0.9560 - auc: 0.9792 - val_loss: 0.
4676 - val_tp: 2601966.0000 - val_fp: 397290.0000 - val_tn: 12006863.0000 - val_fn: 722521.0000 - val_precision: 0.8675 - val_recall: 0.7827 - val_accuracy: 0.9288 - val_auc: 0.9455
Epoch 17/40
2100/2100 [==============================] - 2550s 1s/step - loss: 0.2894 - tp: 21593576.0000 - fp: 2107461.0000 - tn: 77151016.0000 - fn: 2367164.0000 - precision: 0.9111 - recall: 0.9012 - accuracy: 0.9566 - auc: 0.9796 - val_loss: 0.
5073 - val_tp: 2723777.0000 - val_fp: 267464.0000 - val_tn: 11851181.0000 - val_fn: 886218.0000 - val_precision: 0.9106 - val_recall: 0.7545 - val_accuracy: 0.9267 - val_auc: 0.9421
Epoch 18/40
2100/2100 [==============================] - 2565s 1s/step - loss: 0.2915 - tp: 21220710.0000 - fp: 2089500.0000 - tn: 77529384.0000 - fn: 2379606.0000 - precision: 0.9104 - recall: 0.8992 - accuracy: 0.9567 - auc: 0.9792 - val_loss: 0.
4440 - val_tp: 3014233.0000 - val_fp: 194294.0000 - val_tn: 11773351.0000 - val_fn: 746762.0000 - val_precision: 0.9394 - val_recall: 0.8014 - val_accuracy: 0.9402 - val_auc: 0.9537
Epoch 19/40
2100/2100 [==============================] - 2555s 1s/step - loss: 0.2859 - tp: 21497494.0000 - fp: 2128539.0000 - tn: 77269760.0000 - fn: 2323349.0000 - precision: 0.9099 - recall: 0.9025 - accuracy: 0.9569 - auc: 0.9798 - val_loss: 0.
4337 - val_tp: 2842516.0000 - val_fp: 264846.0000 - val_tn: 11878934.0000 - val_fn: 742344.0000 - val_precision: 0.9148 - val_recall: 0.7929 - val_accuracy: 0.9360 - val_auc: 0.9550
Epoch 20/40
2100/2100 [==============================] - 2618s 1s/step - loss: 0.2902 - tp: 21548910.0000 - fp: 2102526.0000 - tn: 77221952.0000 - fn: 2345775.0000 - precision: 0.9111 - recall: 0.9018 - accuracy: 0.9569 - auc: 0.9791 - val_loss: 0.
5074 - val_tp: 2805182.0000 - val_fp: 105571.0000 - val_tn: 11861966.0000 - val_fn: 955921.0000 - val_precision: 0.9637 - val_recall: 0.7458 - val_accuracy: 0.9325 - val_auc: 0.9465
Epoch 21/40
2100/2100 [==============================] - 2633s 1s/step - loss: 0.2838 - tp: 21355176.0000 - fp: 2103115.0000 - tn: 77459040.0000 - fn: 2301871.0000 - precision: 0.9103 - recall: 0.9027 - accuracy: 0.9573 - auc: 0.9797 - val_loss: 0.
5292 - val_tp: 2876840.0000 - val_fp: 499247.0000 - val_tn: 11594560.0000 - val_fn: 757993.0000 - val_precision: 0.8521 - val_recall: 0.7915 - val_accuracy: 0.9201 - val_auc: 0.9410
Epoch 22/40
2100/2100 [==============================] - 2623s 1s/step - loss: 0.2770 - tp: 21105648.0000 - fp: 2105094.0000 - tn: 77782320.0000 - fn: 2226166.0000 - precision: 0.9093 - recall: 0.9046 - accuracy: 0.9580 - auc: 0.9804 - val_loss: 0.
4563 - val_tp: 2796040.0000 - val_fp: 290458.0000 - val_tn: 11870456.0000 - val_fn: 771686.0000 - val_precision: 0.9059 - val_recall: 0.7837 - val_accuracy: 0.9325 - val_auc: 0.9511
Epoch 23/40
2100/2100 [==============================] - 2604s 1s/step - loss: 0.2779 - tp: 21230712.0000 - fp: 2051525.0000 - tn: 77688872.0000 - fn: 2248134.0000 - precision: 0.9119 - recall: 0.9042 - accuracy: 0.9583 - auc: 0.9800 - val_loss: 0.
4772 - val_tp: 2839169.0000 - val_fp: 344193.0000 - val_tn: 11784262.0000 - val_fn: 761016.0000 - val_precision: 0.8919 - val_recall: 0.7886 - val_accuracy: 0.9297 - val_auc: 0.9486
Epoch 24/40
2100/2100 [==============================] - 2607s 1s/step - loss: 0.2831 - tp: 21599932.0000 - fp: 2114749.0000 - tn: 77206248.0000 - fn: 2298204.0000 - precision: 0.9108 - recall: 0.9038 - accuracy: 0.9572 - auc: 0.9799 - val_loss: 0.
4515 - val_tp: 3000318.0000 - val_fp: 428123.0000 - val_tn: 11603006.0000 - val_fn: 697193.0000 - val_precision: 0.8751 - val_recall: 0.8114 - val_accuracy: 0.9285 - val_auc: 0.9539
Epoch 25/40
2100/2100 [==============================] - 2608s 1s/step - loss: 0.2802 - tp: 21684768.0000 - fp: 2090639.0000 - tn: 77166856.0000 - fn: 2276865.0000 - precision: 0.9121 - recall: 0.9050 - accuracy: 0.9577 - auc: 0.9803 - val_loss: 0.
4666 - val_tp: 2881923.0000 - val_fp: 449549.0000 - val_tn: 11719217.0000 - val_fn: 677951.0000 - val_precision: 0.8651 - val_recall: 0.8096 - val_accuracy: 0.9283 - val_auc: 0.9492
Epoch 26/40
2100/2100 [==============================] - 2601s 1s/step - loss: 0.2817 - tp: 21424128.0000 - fp: 2071632.0000 - tn: 77450872.0000 - fn: 2272584.0000 - precision: 0.9118 - recall: 0.9041 - accuracy: 0.9579 - auc: 0.9798 - val_loss: 0.
4771 - val_tp: 2777236.0000 - val_fp: 523426.0000 - val_tn: 11747034.0000 - val_fn: 680944.0000 - val_precision: 0.8414 - val_recall: 0.8031 - val_accuracy: 0.9234 - val_auc: 0.9473
Epoch 27/40
2100/2100 [==============================] - 2557s 1s/step - loss: 0.2761 - tp: 21149424.0000 - fp: 2050887.0000 - tn: 77759664.0000 - fn: 2259208.0000 - precision: 0.9116 - recall: 0.9035 - accuracy: 0.9582 - auc: 0.9802 - val_loss: 0.
4809 - val_tp: 2766963.0000 - val_fp: 356057.0000 - val_tn: 11830343.0000 - val_fn: 775277.0000 - val_precision: 0.8860 - val_recall: 0.7811 - val_accuracy: 0.9281 - val_auc: 0.9473
Epoch 28/40
2100/2100 [==============================] - 2572s 1s/step - loss: 0.2787 - tp: 21614004.0000 - fp: 2105764.0000 - tn: 77265320.0000 - fn: 2234175.0000 - precision: 0.9112 - recall: 0.9063 - accuracy: 0.9580 - auc: 0.9804 - val_loss: 0.
4297 - val_tp: 2776882.0000 - val_fp: 304747.0000 - val_tn: 11905345.0000 - val_fn: 741666.0000 - val_precision: 0.9011 - val_recall: 0.7892 - val_accuracy: 0.9335 - val_auc: 0.9558
Epoch 29/40
2100/2100 [==============================] - 2592s 1s/step - loss: 0.2759 - tp: 21330060.0000 - fp: 2032687.0000 - tn: 77600192.0000 - fn: 2256365.0000 - precision: 0.9130 - recall: 0.9043 - accuracy: 0.9584 - auc: 0.9805 - val_loss: 0.
4579 - val_tp: 2999477.0000 - val_fp: 494545.0000 - val_tn: 11578967.0000 - val_fn: 655651.0000 - val_precision: 0.8585 - val_recall: 0.8206 - val_accuracy: 0.9269 - val_auc: 0.9534
Epoch 30/40
2100/2100 [==============================] - 2581s 1s/step - loss: 0.2724 - tp: 21130112.0000 - fp: 2033978.0000 - tn: 77848336.0000 - fn: 2206652.0000 - precision: 0.9122 - recall: 0.9054 - accuracy: 0.9589 - auc: 0.9804 - val_loss: 0.
4905 - val_tp: 2843808.0000 - val_fp: 315711.0000 - val_tn: 11777024.0000 - val_fn: 792097.0000 - val_precision: 0.9001 - val_recall: 0.7821 - val_accuracy: 0.9296 - val_auc: 0.9485
Epoch 31/40
2100/2100 [==============================] - 2600s 1s/step - loss: 0.2785 - tp: 21125680.0000 - fp: 2037613.0000 - tn: 77795080.0000 - fn: 2260802.0000 - precision: 0.9120 - recall: 0.9033 - accuracy: 0.9584 - auc: 0.9796 - val_loss: 0.
4386 - val_tp: 2841391.0000 - val_fp: 466262.0000 - val_tn: 11790714.0000 - val_fn: 630273.0000 - val_precision: 0.8590 - val_recall: 0.8185 - val_accuracy: 0.9303 - val_auc: 0.9549
Epoch 32/40
2100/2100 [==============================] - 2569s 1s/step - loss: 0.2780 - tp: 21701982.0000 - fp: 2120334.0000 - tn: 77174816.0000 - fn: 2221998.0000 - precision: 0.9110 - recall: 0.9071 - accuracy: 0.9579 - auc: 0.9802 - val_loss: 0.
4458 - val_tp: 2760528.0000 - val_fp: 256875.0000 - val_tn: 11974282.0000 - val_fn: 736955.0000 - val_precision: 0.9149 - val_recall: 0.7893 - val_accuracy: 0.9368 - val_auc: 0.9520
Epoch 33/40
2100/2100 [==============================] - 2558s 1s/step - loss: 0.2798 - tp: 21431360.0000 - fp: 2104855.0000 - tn: 77436496.0000 - fn: 2246416.0000 - precision: 0.9106 - recall: 0.9051 - accuracy: 0.9578 - auc: 0.9799 - val_loss: 0.
4787 - val_tp: 2902400.0000 - val_fp: 434973.0000 - val_tn: 11637810.0000 - val_fn: 753457.0000 - val_precision: 0.8697 - val_recall: 0.7939 - val_accuracy: 0.9244 - val_auc: 0.9494
Epoch 34/40
2100/2100 [==============================] - 2540s 1s/step - loss: 0.2773 - tp: 21209216.0000 - fp: 2046492.0000 - tn: 77703984.0000 - fn: 2259570.0000 - precision: 0.9120 - recall: 0.9037 - accuracy: 0.9583 - auc: 0.9800 - val_loss: 0.
4282 - val_tp: 2892899.0000 - val_fp: 378247.0000 - val_tn: 11801593.0000 - val_fn: 655901.0000 - val_precision: 0.8844 - val_recall: 0.8152 - val_accuracy: 0.9343 - val_auc: 0.9564
Epoch 35/40
2100/2100 [==============================] - 2548s 1s/step - loss: 0.2784 - tp: 21515622.0000 - fp: 2084525.0000 - tn: 77370184.0000 - fn: 2248908.0000 - precision: 0.9117 - recall: 0.9054 - accuracy: 0.9580 - auc: 0.9804 - val_loss: 0.
4553 - val_tp: 3086603.0000 - val_fp: 467495.0000 - val_tn: 11473592.0000 - val_fn: 700950.0000 - val_precision: 0.8685 - val_recall: 0.8149 - val_accuracy: 0.9257 - val_auc: 0.9544
Epoch 36/40
2100/2100 [==============================] - 2541s 1s/step - loss: 0.2708 - tp: 20892576.0000 - fp: 2005985.0000 - tn: 78121936.0000 - fn: 2198694.0000 - precision: 0.9124 - recall: 0.9048 - accuracy: 0.9593 - auc: 0.9805 - val_loss: 0.
4393 - val_tp: 2703510.0000 - val_fp: 394771.0000 - val_tn: 11969160.0000 - val_fn: 661199.0000 - val_precision: 0.8726 - val_recall: 0.8035 - val_accuracy: 0.9329 - val_auc: 0.9538
Epoch 37/40
2100/2100 [==============================] - 2552s 1s/step - loss: 0.2788 - tp: 21802620.0000 - fp: 2101735.0000 - tn: 77067912.0000 - fn: 2246918.0000 - precision: 0.9121 - recall: 0.9066 - accuracy: 0.9579 - auc: 0.9805 - val_loss: 0.
4474 - val_tp: 2806677.0000 - val_fp: 219556.0000 - val_tn: 11887825.0000 - val_fn: 814582.0000 - val_precision: 0.9274 - val_recall: 0.7751 - val_accuracy: 0.9343 - val_auc: 0.9537
Epoch 38/40
2100/2100 [==============================] - 2545s 1s/step - loss: 0.2769 - tp: 21191620.0000 - fp: 2046041.0000 - tn: 77717744.0000 - fn: 2263741.0000 - precision: 0.9120 - recall: 0.9035 - accuracy: 0.9582 - auc: 0.9801 - val_loss: 0.
4414 - val_tp: 2927919.0000 - val_fp: 286995.0000 - val_tn: 11783958.0000 - val_fn: 729768.0000 - val_precision: 0.9107 - val_recall: 0.8005 - val_accuracy: 0.9354 - val_auc: 0.9544
Epoch 39/40
2100/2100 [==============================] - 2545s 1s/step - loss: 0.2785 - tp: 21171298.0000 - fp: 2047544.0000 - tn: 77738952.0000 - fn: 2261357.0000 - precision: 0.9118 - recall: 0.9035 - accuracy: 0.9583 - auc: 0.9798 - val_loss: 0.
4576 - val_tp: 2779792.0000 - val_fp: 346642.0000 - val_tn: 11869174.0000 - val_fn: 733032.0000 - val_precision: 0.8891 - val_recall: 0.7913 - val_accuracy: 0.9314 - val_auc: 0.9517
Epoch 40/40
2100/2100 [==============================] - 2538s 1s/step - loss: 0.2811 - tp: 21804190.0000 - fp: 2110823.0000 - tn: 77014800.0000 - fn: 2289448.0000 - precision: 0.9117 - recall: 0.9050 - accuracy: 0.9574 - auc: 0.9803 - val_loss: 0.
4445 - val_tp: 2933937.0000 - val_fp: 327429.0000 - val_tn: 11730665.0000 - val_fn: 736609.0000 - val_precision: 0.8996 - val_recall: 0.7993 - val_accuracy: 0.9324 - val_auc: 0.9543
667/667 [==============================] - 163s 244ms/step - loss: 0.5248 - tp: 6207113.0000 - fp: 1521051.0000 - tn: 23569214.0000 - fn: 1470623.0000 - precision: 0.8032 - recall: 0.8085 - accuracy: 0.9087 - auc: 0.9439
2020/06/21 05:32:13 INFO mlflow.projects: === Run (ID '1e98b3ed2e1d421da9592058bd5587a8') succeeded ===

2.1.15 git exp 4

git status
git log -1
(tensorflow_nightly) [ye53nis@node161 drmed-git]$ git status
git log -1
# On branch exp-310520-unet
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#   ...
nothing added to commit but untracked files present (use "git add" to track)

(tensorflow_nightly) [ye53nis@node161 drmed-git]$ git log -1
commit 8adaca82d8e092997828338bc2b730568c3ff74b
Author: Alex Seltmann <seltmann@posteo.de>
Date:   Mon Jun 22 00:45:48 2020 +0200

    exp-310520-unet run 3 bs=3

2.1.16 experimental run 4 - bs=7

mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src -P batch_size=7 -P epochs=40 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact_Sep2019 -P steps_per_epoch=910 -P validation_steps=225
2020/06/22 01:46:18 INFO mlflow.projects: === Created directory /tmp/tmpbqom5l09 for downloading remote URIs passed to arguments of type 'path' ===
2020/06/22 01:46:18 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/fluotracify/training/train.py /
beegfs/ye53nis/drmed-git/src 7 0.2 16384 None 40 /beegfs/ye53nis/saves/firstartefact_Sep2019 910 225' in run with ID '306234c75c9c48058cbd694579eff31b' ===
2020-06-22 01:46:38.397450: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-06-22 01:46:38.397497: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favour of importlib; see the modul
e's documentation for alternative uses
  import imp
2.3.0-dev20200527
2020-06-22 01:47:08.147584: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-06-22 01:47:08.147650: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-06-22 01:47:08.147696: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node161): /proc/driver/nvidia/version does not exist
GPUs:  []
train 0 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set027.csv
train 1 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set087.csv
train 2 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set003.csv
train 3 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set056.csv
train 4 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set076.csv
train 5 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set094.csv
train 6 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set017.csv
train 7 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set074.csv
train 8 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set055.csv
train 9 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set096.csv
train 10 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set054.csv
train 11 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set093.csv
train 12 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set079.csv
train 13 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set014.csv
train 14 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set008.csv
train 15 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set031.csv
train 16 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set023.csv
train 17 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set025.csv
train 18 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set034.csv
train 19 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set009.csv
train 20 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set044.csv
train 21 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set063.csv
train 22 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set004.csv
train 23 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set072.csv
train 24 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set046.csv
train 25 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set049.csv
train 26 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set007.csv
train 27 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set100.csv
train 28 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set083.csv
train 29 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set077.csv
train 30 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set061.csv
train 31 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set081.csv
train 32 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set091.csv
train 33 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set069.csv
train 34 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set052.csv
train 35 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set028.csv
train 36 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set019.csv
train 37 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set057.csv
train 38 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set064.csv
train 39 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set075.csv
train 40 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set002.csv
train 41 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set062.csv
train 42 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set043.csv
train 43 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set042.csv
train 44 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set005.csv
train 45 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set016.csv
train 46 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set018.csv
train 47 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set041.csv
train 48 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set039.csv
train 49 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set006.csv
train 50 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set092.csv
train 51 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set060.csv
train 52 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set001.csv
train 53 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set035.csv
train 54 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set029.csv
train 55 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set051.csv
train 56 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set012.csv
train 57 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set036.csv
train 58 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set024.csv
train 59 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set053.csv
train 60 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set011.csv
train 61 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set032.csv
train 62 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set067.csv
train 63 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set058.csv
train 64 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set080.csv
train 65 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set086.csv
train 66 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set033.csv
train 67 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set085.csv
train 68 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set015.csv
train 69 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set090.csv
train 70 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set020.csv
train 71 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set030.csv
train 72 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set050.csv
train 73 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set098.csv
train 74 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set099.csv
train 75 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set070.csv
train 76 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set021.csv
train 77 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set095.csv
train 78 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set073.csv
train 79 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set078.csv
test 80 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set026.csv
test 81 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set038.csv
test 82 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set082.csv
test 83 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set047.csv
test 84 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set040.csv
test 85 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set066.csv
test 86 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set059.csv
test 87 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set013.csv
test 88 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set089.csv
test 89 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set071.csv
test 90 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set088.csv
test 91 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set037.csv
test 92 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set022.csv
test 93 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set084.csv
test 94 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set010.csv
test 95 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set097.csv
test 96 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set068.csv
test 97 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set065.csv
test 98 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set048.csv
test 99 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set045.csv
shapes of feature dataframe: (20000, 8000) and label dataframe: (20000, 8000)
shapes of feature dataframe: (20000, 2000) and label dataframe: (20000, 2000)

for each 20,000 timestap trace there are the following numbers of corrupted timesteps:
 label001_1    6286
label001_1    2568
label001_1    4495
label001_1    4414
label001_1    1105
dtype: int64
2020-06-22 01:52:37.438938: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-06-22 01:52:37.449993: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2300000000 Hz
2020-06-22 01:52:37.451959: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5592ec827380 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-22 01:52:37.451991: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
number of training examples: 6400, number of validation examples: 1600

------------------------
number of test examples: 2000

input - shape:   (None, 16384, 1)
output - shape:  (None, 16384, 1)
2020-06-22 01:52:41.793790: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the ABCs from 'collections' instead
 of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  if not isinstance(wrapped_dict, collections.Mapping):
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature()
 or inspect.getfullargspec()
  all_param_names, _, _, all_default_values = inspect.getargspec(fn)  # pylint: disable=W1505
Epoch 1/40
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from
'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  if not isinstance(values, collections.Sequence):
  1/910 [..............................] - ETA: 0s - loss: 1.5123 - tp: 3489.0000 - fp: 12945.0000 - tn: 73461.0000 - fn: 24793.0000 - precision: 0.2123 - recall: 0.1234 - accuracy: 0.6710 - auc: 0.49312020-06-22 01:52:55.410673: I tens
orflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager.profiler) is deprecated and w
ill be removed after 2020-07-01.
Instructions for updating:
use `tf.profiler.experimental.stop` instead.
2020-06-22 01:52:57.883904: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_06_22_01_52_57
2020-06-22 01:52:57.900920: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_06_22_01_52_57/node161.trace.json.gz
2020-06-22 01:52:57.938257: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_06_22_01_52_57
2020-06-22 01:52:57.938413: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_06_22_01_52_57/node161.memory_profile.json.gz
2020-06-22 01:52:57.941154: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_06_22_01_52_57Dumped tool data for xplane.pb to /tmp/tb/train/plugins/profile/2020_06_22_0
1_52_57/node161.xplane.pb
Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_06_22_01_52_57/node161.overview_page.pb
Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_06_22_01_52_57/node161.input_pipeline.pb
Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_06_22_01_52_57/node161.tensorflow_stats.pb
Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_06_22_01_52_57/node161.kernel_stats.pb

910/910 [==============================] - 2234s 2s/step - loss: 1.1284 - tp: 16644571.0000 - fp: 5060115.0000 - tn: 75211888.0000 - fn: 7449514.0000 - precision: 0.7669 - recall: 0.6908 - accuracy: 0.8801 - auc: 0.9092 - val_loss: 44.1
496 - val_tp: 5991771.0000 - val_fp: 19810554.0000 - val_tn: 2354.0000 - val_fn: 117.0000 - val_precision: 0.2322 - val_recall: 1.0000 - val_accuracy: 0.2323 - val_auc: 0.5235
Epoch 2/40
910/910 [==============================] - 2228s 2s/step - loss: 0.6506 - tp: 18676100.0000 - fp: 3725706.0000 - tn: 76768256.0000 - fn: 5196001.0000 - precision: 0.8337 - recall: 0.7823 - accuracy: 0.9145 - auc: 0.9421 - val_loss: 4.02
94 - val_tp: 5567715.0000 - val_fp: 7766004.0000 - val_tn: 12285671.0000 - val_fn: 185410.0000 - val_precision: 0.4176 - val_recall: 0.9678 - val_accuracy: 0.6919 - val_auc: 0.8550
Epoch 3/40
910/910 [==============================] - 2209s 2s/step - loss: 0.4869 - tp: 19773668.0000 - fp: 3339711.0000 - tn: 77156064.0000 - fn: 4096586.0000 - precision: 0.8555 - recall: 0.8284 - accuracy: 0.9287 - auc: 0.9571 - val_loss: 15.0
428 - val_tp: 5964847.0000 - val_fp: 16182339.0000 - val_tn: 3635866.0000 - val_fn: 21748.0000 - val_precision: 0.2693 - val_recall: 0.9964 - val_accuracy: 0.3721 - val_auc: 0.6686
Epoch 4/40
910/910 [==============================] - 2196s 2s/step - loss: 0.4160 - tp: 20489738.0000 - fp: 2947256.0000 - tn: 77507616.0000 - fn: 3421442.0000 - precision: 0.8742 - recall: 0.8569 - accuracy: 0.9390 - auc: 0.9659 - val_loss: 1.94
92 - val_tp: 5844006.0000 - val_fp: 12298817.0000 - val_tn: 7631163.0000 - val_fn: 30814.0000 - val_precision: 0.3221 - val_recall: 0.9948 - val_accuracy: 0.5222 - val_auc: 0.9445
Epoch 5/40
910/910 [==============================] - 2197s 2s/step - loss: 0.3808 - tp: 20651254.0000 - fp: 2753622.0000 - tn: 77775072.0000 - fn: 3186176.0000 - precision: 0.8823 - recall: 0.8663 - accuracy: 0.9431 - auc: 0.9697 - val_loss: 0.48
60 - val_tp: 5068080.0000 - val_fp: 972440.0000 - val_tn: 19094874.0000 - val_fn: 669407.0000 - val_precision: 0.8390 - val_recall: 0.8833 - val_accuracy: 0.9364 - val_auc: 0.9722
Epoch 6/40
910/910 [==============================] - 2186s 2s/step - loss: 0.3788 - tp: 20770170.0000 - fp: 2708911.0000 - tn: 77787896.0000 - fn: 3099100.0000 - precision: 0.8846 - recall: 0.8702 - accuracy: 0.9444 - auc: 0.9703 - val_loss: 0.86
28 - val_tp: 5389513.0000 - val_fp: 2524440.0000 - val_tn: 17394656.0000 - val_fn: 496188.0000 - val_precision: 0.6810 - val_recall: 0.9157 - val_accuracy: 0.8829 - val_auc: 0.9492
Epoch 7/40
910/910 [==============================] - 2187s 2s/step - loss: 0.3621 - tp: 21038052.0000 - fp: 2650316.0000 - tn: 77692112.0000 - fn: 2985530.0000 - precision: 0.8881 - recall: 0.8757 - accuracy: 0.9460 - auc: 0.9724 - val_loss: 0.90
81 - val_tp: 5562172.0000 - val_fp: 3465595.0000 - val_tn: 16430461.0000 - val_fn: 346572.0000 - val_precision: 0.6161 - val_recall: 0.9413 - val_accuracy: 0.8523 - val_auc: 0.9356
Epoch 8/40
910/910 [==============================] - 2193s 2s/step - loss: 0.3553 - tp: 20964808.0000 - fp: 2536411.0000 - tn: 77880080.0000 - fn: 2984777.0000 - precision: 0.8921 - recall: 0.8754 - accuracy: 0.9471 - auc: 0.9726 - val_loss: 1.75
52 - val_tp: 5773672.0000 - val_fp: 6575655.0000 - val_tn: 13365159.0000 - val_fn: 90314.0000 - val_precision: 0.4675 - val_recall: 0.9846 - val_accuracy: 0.7417 - val_auc: 0.9457
Epoch 9/40
910/910 [==============================] - 2191s 2s/step - loss: 0.3477 - tp: 20855856.0000 - fp: 2530233.0000 - tn: 78070912.0000 - fn: 2908993.0000 - precision: 0.8918 - recall: 0.8776 - accuracy: 0.9479 - auc: 0.9732 - val_loss: 5.37
18 - val_tp: 5639818.0000 - val_fp: 10545935.0000 - val_tn: 9573186.0000 - val_fn: 45861.0000 - val_precision: 0.3484 - val_recall: 0.9919 - val_accuracy: 0.5895 - val_auc: 0.8627
Epoch 10/40
910/910 [==============================] - 2198s 2s/step - loss: 0.3453 - tp: 20967864.0000 - fp: 2537372.0000 - tn: 77932464.0000 - fn: 2928327.0000 - precision: 0.8921 - recall: 0.8775 - accuracy: 0.9476 - auc: 0.9739 - val_loss: 6.18
03 - val_tp: 5843714.0000 - val_fp: 6972956.0000 - val_tn: 12840040.0000 - val_fn: 148090.0000 - val_precision: 0.4559 - val_recall: 0.9753 - val_accuracy: 0.7240 - val_auc: 0.8662
Epoch 11/40
910/910 [==============================] - 2199s 2s/step - loss: 0.3386 - tp: 21061526.0000 - fp: 2493371.0000 - tn: 77979232.0000 - fn: 2831932.0000 - precision: 0.8941 - recall: 0.8815 - accuracy: 0.9490 - auc: 0.9744 - val_loss: 1.93
77 - val_tp: 5141006.0000 - val_fp: 5024287.0000 - val_tn: 14889905.0000 - val_fn: 749602.0000 - val_precision: 0.5057 - val_recall: 0.8727 - val_accuracy: 0.7762 - val_auc: 0.8851
Epoch 12/40
910/910 [==============================] - 2189s 2s/step - loss: 0.3032 - tp: 21443394.0000 - fp: 2244059.0000 - tn: 78197192.0000 - fn: 2481465.0000 - precision: 0.9053 - recall: 0.8963 - accuracy: 0.9547 - auc: 0.9782 - val_loss: 0.41
02 - val_tp: 4447303.0000 - val_fp: 81983.0000 - val_tn: 19938508.0000 - val_fn: 1337008.0000 - val_precision: 0.9819 - val_recall: 0.7689 - val_accuracy: 0.9450 - val_auc: 0.9689
Epoch 13/40
910/910 [==============================] - 2190s 2s/step - loss: 0.2934 - tp: 21384318.0000 - fp: 2224470.0000 - tn: 78382704.0000 - fn: 2374534.0000 - precision: 0.9058 - recall: 0.9001 - accuracy: 0.9559 - auc: 0.9790 - val_loss: 0.39
33 - val_tp: 4709935.0000 - val_fp: 185454.0000 - val_tn: 19874024.0000 - val_fn: 1035376.0000 - val_precision: 0.9621 - val_recall: 0.8198 - val_accuracy: 0.9527 - val_auc: 0.9764
Epoch 14/40
910/910 [==============================] - 2191s 2s/step - loss: 0.2891 - tp: 21659198.0000 - fp: 2221361.0000 - tn: 78137264.0000 - fn: 2348280.0000 - precision: 0.9070 - recall: 0.9022 - accuracy: 0.9562 - auc: 0.9795 - val_loss: 0.34
77 - val_tp: 4962272.0000 - val_fp: 238349.0000 - val_tn: 19698052.0000 - val_fn: 906131.0000 - val_precision: 0.9542 - val_recall: 0.8456 - val_accuracy: 0.9556 - val_auc: 0.9784
Epoch 15/40
910/910 [==============================] - 2190s 2s/step - loss: 0.2855 - tp: 21305984.0000 - fp: 2168209.0000 - tn: 78570984.0000 - fn: 2320894.0000 - precision: 0.9076 - recall: 0.9018 - accuracy: 0.9570 - auc: 0.9794 - val_loss: 0.42
92 - val_tp: 5448927.0000 - val_fp: 691788.0000 - val_tn: 19190140.0000 - val_fn: 473948.0000 - val_precision: 0.8873 - val_recall: 0.9200 - val_accuracy: 0.9548 - val_auc: 0.9843
Epoch 16/40
910/910 [==============================] - 2206s 2s/step - loss: 0.2860 - tp: 21768336.0000 - fp: 2197317.0000 - tn: 78071472.0000 - fn: 2328913.0000 - precision: 0.9083 - recall: 0.9034 - accuracy: 0.9566 - auc: 0.9798 - val_loss: 0.38
76 - val_tp: 5178300.0000 - val_fp: 782337.0000 - val_tn: 19316320.0000 - val_fn: 527843.0000 - val_precision: 0.8687 - val_recall: 0.9075 - val_accuracy: 0.9492 - val_auc: 0.9825
Epoch 17/40
910/910 [==============================] - 2195s 2s/step - loss: 0.2846 - tp: 21434488.0000 - fp: 2169029.0000 - tn: 78438552.0000 - fn: 2324017.0000 - precision: 0.9081 - recall: 0.9022 - accuracy: 0.9570 - auc: 0.9797 - val_loss: 0.30
14 - val_tp: 5075497.0000 - val_fp: 245322.0000 - val_tn: 19617398.0000 - val_fn: 866585.0000 - val_precision: 0.9539 - val_recall: 0.8542 - val_accuracy: 0.9569 - val_auc: 0.9784
Epoch 18/40
910/910 [==============================] - 2194s 2s/step - loss: 0.2779 - tp: 21615928.0000 - fp: 2131081.0000 - tn: 78362064.0000 - fn: 2257103.0000 - precision: 0.9103 - recall: 0.9055 - accuracy: 0.9580 - auc: 0.9802 - val_loss: 0.35
91 - val_tp: 5315886.0000 - val_fp: 630662.0000 - val_tn: 19377184.0000 - val_fn: 481064.0000 - val_precision: 0.8939 - val_recall: 0.9170 - val_accuracy: 0.9569 - val_auc: 0.9853
Epoch 19/40
910/910 [==============================] - 2199s 2s/step - loss: 0.2810 - tp: 21823838.0000 - fp: 2154952.0000 - tn: 78086368.0000 - fn: 2300819.0000 - precision: 0.9101 - recall: 0.9046 - accuracy: 0.9573 - auc: 0.9804 - val_loss: 0.29
36 - val_tp: 5247304.0000 - val_fp: 327869.0000 - val_tn: 19475176.0000 - val_fn: 754455.0000 - val_precision: 0.9412 - val_recall: 0.8743 - val_accuracy: 0.9581 - val_auc: 0.9805
Epoch 20/40
910/910 [==============================] - 2192s 2s/step - loss: 0.2764 - tp: 21738176.0000 - fp: 2123769.0000 - tn: 78235688.0000 - fn: 2268399.0000 - precision: 0.9110 - recall: 0.9055 - accuracy: 0.9579 - auc: 0.9807 - val_loss: 0.31
80 - val_tp: 5172404.0000 - val_fp: 301019.0000 - val_tn: 19568382.0000 - val_fn: 762993.0000 - val_precision: 0.9450 - val_recall: 0.8715 - val_accuracy: 0.9588 - val_auc: 0.9816
Epoch 21/40
910/910 [==============================] - 2198s 2s/step - loss: 0.2753 - tp: 21565306.0000 - fp: 2109464.0000 - tn: 78441872.0000 - fn: 2249478.0000 - precision: 0.9109 - recall: 0.9055 - accuracy: 0.9582 - auc: 0.9805 - val_loss: 0.32
99 - val_tp: 4898108.0000 - val_fp: 133276.0000 - val_tn: 19742242.0000 - val_fn: 1031169.0000 - val_precision: 0.9735 - val_recall: 0.8261 - val_accuracy: 0.9549 - val_auc: 0.9758
Epoch 22/40
910/910 [==============================] - 2194s 2s/step - loss: 0.2705 - tp: 21872064.0000 - fp: 2096978.0000 - tn: 78186784.0000 - fn: 2210292.0000 - precision: 0.9125 - recall: 0.9082 - accuracy: 0.9587 - auc: 0.9812 - val_loss: 0.32
48 - val_tp: 5379907.0000 - val_fp: 361207.0000 - val_tn: 19421612.0000 - val_fn: 642071.0000 - val_precision: 0.9371 - val_recall: 0.8934 - val_accuracy: 0.9611 - val_auc: 0.9833
Epoch 23/40
910/910 [==============================] - 2192s 2s/step - loss: 0.2693 - tp: 21757518.0000 - fp: 2089232.0000 - tn: 78320768.0000 - fn: 2198508.0000 - precision: 0.9124 - recall: 0.9082 - accuracy: 0.9589 - auc: 0.9810 - val_loss: 0.29
60 - val_tp: 5305212.0000 - val_fp: 364327.0000 - val_tn: 19508656.0000 - val_fn: 626605.0000 - val_precision: 0.9357 - val_recall: 0.8944 - val_accuracy: 0.9616 - val_auc: 0.9831
Epoch 24/40
910/910 [==============================] - 2182s 2s/step - loss: 0.2644 - tp: 21582250.0000 - fp: 2055042.0000 - tn: 78581192.0000 - fn: 2147551.0000 - precision: 0.9131 - recall: 0.9095 - accuracy: 0.9597 - auc: 0.9816 - val_loss: 0.31
56 - val_tp: 5259415.0000 - val_fp: 362064.0000 - val_tn: 19527056.0000 - val_fn: 656259.0000 - val_precision: 0.9356 - val_recall: 0.8891 - val_accuracy: 0.9605 - val_auc: 0.9826
Epoch 25/40
910/910 [==============================] - 2182s 2s/step - loss: 0.2646 - tp: 21752094.0000 - fp: 2060925.0000 - tn: 78391832.0000 - fn: 2161242.0000 - precision: 0.9135 - recall: 0.9096 - accuracy: 0.9595 - auc: 0.9816 - val_loss: 0.30
17 - val_tp: 5312792.0000 - val_fp: 398301.0000 - val_tn: 19491284.0000 - val_fn: 602422.0000 - val_precision: 0.9303 - val_recall: 0.8982 - val_accuracy: 0.9612 - val_auc: 0.9832
Epoch 26/40
910/910 [==============================] - 2111s 2s/step - loss: 0.2703 - tp: 21731022.0000 - fp: 2103277.0000 - tn: 78346144.0000 - fn: 2185602.0000 - precision: 0.9118 - recall: 0.9086 - accuracy: 0.9589 - auc: 0.9811 - val_loss: 0.31
48 - val_tp: 5491565.0000 - val_fp: 401219.0000 - val_tn: 19289576.0000 - val_fn: 622446.0000 - val_precision: 0.9319 - val_recall: 0.8982 - val_accuracy: 0.9603 - val_auc: 0.9844
Epoch 27/40
910/910 [==============================] - 2066s 2s/step - loss: 0.2638 - tp: 21622916.0000 - fp: 2036434.0000 - tn: 78564656.0000 - fn: 2142006.0000 - precision: 0.9139 - recall: 0.9099 - accuracy: 0.9600 - auc: 0.9815 - val_loss: 0.28
12 - val_tp: 5310123.0000 - val_fp: 347345.0000 - val_tn: 19514636.0000 - val_fn: 632692.0000 - val_precision: 0.9386 - val_recall: 0.8935 - val_accuracy: 0.9620 - val_auc: 0.9826
Epoch 28/40
910/910 [==============================] - 2084s 2s/step - loss: 0.2644 - tp: 21746928.0000 - fp: 2054824.0000 - tn: 78423840.0000 - fn: 2140490.0000 - precision: 0.9137 - recall: 0.9104 - accuracy: 0.9598 - auc: 0.9815 - val_loss: 0.29
02 - val_tp: 5105988.0000 - val_fp: 301224.0000 - val_tn: 19705344.0000 - val_fn: 692246.0000 - val_precision: 0.9443 - val_recall: 0.8806 - val_accuracy: 0.9615 - val_auc: 0.9816
Epoch 29/40
910/910 [==============================] - 2091s 2s/step - loss: 0.2657 - tp: 21903456.0000 - fp: 2055350.0000 - tn: 78244200.0000 - fn: 2163118.0000 - precision: 0.9142 - recall: 0.9101 - accuracy: 0.9596 - auc: 0.9816 - val_loss: 0.29
28 - val_tp: 5111563.0000 - val_fp: 355341.0000 - val_tn: 19707988.0000 - val_fn: 629909.0000 - val_precision: 0.9350 - val_recall: 0.8903 - val_accuracy: 0.9618 - val_auc: 0.9825
Epoch 30/40
910/910 [==============================] - 2097s 2s/step - loss: 0.2663 - tp: 22007504.0000 - fp: 2072930.0000 - tn: 78112200.0000 - fn: 2173470.0000 - precision: 0.9139 - recall: 0.9101 - accuracy: 0.9593 - auc: 0.9816 - val_loss: 0.30
39 - val_tp: 5361046.0000 - val_fp: 369167.0000 - val_tn: 19426364.0000 - val_fn: 648220.0000 - val_precision: 0.9356 - val_recall: 0.8921 - val_accuracy: 0.9606 - val_auc: 0.9831
Epoch 31/40
910/910 [==============================] - 2092s 2s/step - loss: 0.2670 - tp: 21883968.0000 - fp: 2114417.0000 - tn: 78182464.0000 - fn: 2185257.0000 - precision: 0.9119 - recall: 0.9092 - accuracy: 0.9588 - auc: 0.9816 - val_loss: 0.29
74 - val_tp: 5307833.0000 - val_fp: 308903.0000 - val_tn: 19507264.0000 - val_fn: 680801.0000 - val_precision: 0.9450 - val_recall: 0.8863 - val_accuracy: 0.9616 - val_auc: 0.9830
Epoch 32/40
910/910 [==============================] - 2168s 2s/step - loss: 0.2678 - tp: 21697564.0000 - fp: 2002839.0000 - tn: 78462496.0000 - fn: 2203202.0000 - precision: 0.9155 - recall: 0.9078 - accuracy: 0.9597 - auc: 0.9811 - val_loss: 0.29
81 - val_tp: 4991380.0000 - val_fp: 318033.0000 - val_tn: 19848366.0000 - val_fn: 647024.0000 - val_precision: 0.9401 - val_recall: 0.8852 - val_accuracy: 0.9626 - val_auc: 0.9819
Epoch 33/40
910/910 [==============================] - 2143s 2s/step - loss: 0.2641 - tp: 21608556.0000 - fp: 2029292.0000 - tn: 78569336.0000 - fn: 2158886.0000 - precision: 0.9142 - recall: 0.9092 - accuracy: 0.9599 - auc: 0.9815 - val_loss: 0.30
89 - val_tp: 5377773.0000 - val_fp: 383774.0000 - val_tn: 19419054.0000 - val_fn: 624197.0000 - val_precision: 0.9334 - val_recall: 0.8960 - val_accuracy: 0.9609 - val_auc: 0.9835
Epoch 34/40
910/910 [==============================] - 2148s 2s/step - loss: 0.2636 - tp: 21989192.0000 - fp: 2079253.0000 - tn: 78164408.0000 - fn: 2133219.0000 - precision: 0.9136 - recall: 0.9116 - accuracy: 0.9596 - auc: 0.9818 - val_loss: 0.28
72 - val_tp: 5242507.0000 - val_fp: 383622.0000 - val_tn: 19573328.0000 - val_fn: 605341.0000 - val_precision: 0.9318 - val_recall: 0.8965 - val_accuracy: 0.9617 - val_auc: 0.9836
Epoch 35/40
910/910 [==============================] - 2156s 2s/step - loss: 0.2642 - tp: 21771006.0000 - fp: 2059886.0000 - tn: 78393632.0000 - fn: 2141526.0000 - precision: 0.9136 - recall: 0.9104 - accuracy: 0.9597 - auc: 0.9816 - val_loss: 0.30
80 - val_tp: 5192984.0000 - val_fp: 352885.0000 - val_tn: 19633610.0000 - val_fn: 625321.0000 - val_precision: 0.9364 - val_recall: 0.8925 - val_accuracy: 0.9621 - val_auc: 0.9832
Epoch 36/40
910/910 [==============================] - 2114s 2s/step - loss: 0.2641 - tp: 21936120.0000 - fp: 2076747.0000 - tn: 78200080.0000 - fn: 2153085.0000 - precision: 0.9135 - recall: 0.9106 - accuracy: 0.9595 - auc: 0.9818 - val_loss: 0.29
62 - val_tp: 5151439.0000 - val_fp: 372261.0000 - val_tn: 19674524.0000 - val_fn: 606575.0000 - val_precision: 0.9326 - val_recall: 0.8947 - val_accuracy: 0.9621 - val_auc: 0.9835
Epoch 37/40
910/910 [==============================] - 2083s 2s/step - loss: 0.2591 - tp: 21745408.0000 - fp: 2014129.0000 - tn: 78506376.0000 - fn: 2100146.0000 - precision: 0.9152 - recall: 0.9119 - accuracy: 0.9606 - auc: 0.9821 - val_loss: 0.28
98 - val_tp: 5332435.0000 - val_fp: 349131.0000 - val_tn: 19493176.0000 - val_fn: 630050.0000 - val_precision: 0.9386 - val_recall: 0.8943 - val_accuracy: 0.9621 - val_auc: 0.9841
Epoch 38/40
910/910 [==============================] - 2067s 2s/step - loss: 0.2677 - tp: 22040308.0000 - fp: 2034232.0000 - tn: 78099864.0000 - fn: 2191664.0000 - precision: 0.9155 - recall: 0.9096 - accuracy: 0.9595 - auc: 0.9813 - val_loss: 0.30
58 - val_tp: 5293916.0000 - val_fp: 364251.0000 - val_tn: 19503016.0000 - val_fn: 643618.0000 - val_precision: 0.9356 - val_recall: 0.8916 - val_accuracy: 0.9609 - val_auc: 0.9818
Epoch 39/40
910/910 [==============================] - 2088s 2s/step - loss: 0.2665 - tp: 21782964.0000 - fp: 2071501.0000 - tn: 78364600.0000 - fn: 2146992.0000 - precision: 0.9132 - recall: 0.9103 - accuracy: 0.9596 - auc: 0.9812 - val_loss: 0.28
94 - val_tp: 5190450.0000 - val_fp: 358739.0000 - val_tn: 19624484.0000 - val_fn: 631125.0000 - val_precision: 0.9354 - val_recall: 0.8916 - val_accuracy: 0.9616 - val_auc: 0.9823
Epoch 40/40
910/910 [==============================] - 2093s 2s/step - loss: 0.2610 - tp: 21703476.0000 - fp: 2048057.0000 - tn: 78489664.0000 - fn: 2124881.0000 - precision: 0.9138 - recall: 0.9108 - accuracy: 0.9600 - auc: 0.9819 - val_loss: 0.28
97 - val_tp: 5014376.0000 - val_fp: 339650.0000 - val_tn: 19831600.0000 - val_fn: 619172.0000 - val_precision: 0.9366 - val_recall: 0.8901 - val_accuracy: 0.9628 - val_auc: 0.9828
286/286 [==============================] - 134s 469ms/step - loss: 0.3156 - tp: 6887079.0000 - fp: 534559.0000 - tn: 24555692.0000 - fn: 790657.0000 - precision: 0.9280 - recall: 0.8970 - accuracy: 0.9596 - auc: 0.9821
2020/06/23 01:58:39 INFO mlflow.projects: === Run (ID '306234c75c9c48058cbd694579eff31b') succeeded ===




2.1.17 git exp 5

git status
git log -1
(base) [ye53nis@node011 drmed-git]$ git status
git log -1
# On branch exp-310520-unet
# ...
no changes added to commit (use "git add" and/or "git commit -a")

(base) [ye53nis@node011 drmed-git]$ git log -1
commit 8811f54920c7089b8a27d7f39a50acede5be64c9
Author: Apoplex <oligolex@vivaldi.net>
Date:   Fri Jul 3 00:51:59 2020 +0200

    Incorporate unet prediction in plotting function

2.1.18 experimental run 5 - full dataset, length=2**13=8192

mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src -P batch_size=5 -P length_delimiter=8192 -P epochs=40 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact_Sep2019 -P steps_per_epoch=1280 -P validation_steps=320
(tensorflow_nightly) [ye53nis@node011 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src -P batch_size=5 -P length_delimiter=8192 -P epochs=40 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartefact
_Sep2019 -P steps_per_epoch=1280 -P validation_steps=320
2020/07/03 13:45:30 INFO mlflow.projects: === Created directory /tmp/tmpqf8ewz0u for downloading remote URIs passed to arguments of type 'path' ===
2020/07/03 13:45:30 INFO mlflow.projects: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b 1>&2 && python src/fluotracify/training/train.py /
beegfs/ye53nis/drmed-git/src 5 0.2 8192 None 40 /beegfs/ye53nis/saves/firstartefact_Sep2019 1280 320' in run with ID 'd9b44dc2e3d44ea1a71129808b642af6' ===
2020-07-03 13:45:57.151501: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-07-03 13:45:57.151600: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/load_library.py:23: DeprecationWarning: the imp module is deprecated in favour of importlib; see the modul
e's documentation for alternative uses
  import imp
2.3.0-dev20200527
2020-07-03 13:46:34.363762: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-07-03 13:46:34.363823: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-07-03 13:46:34.363865: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node011): /proc/driver/nvidia/version does not exist
GPUs:  []
train 0 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set027.csv
train 1 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set087.csv
train 2 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set003.csv
train 3 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set056.csv
train 4 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set076.csv
train 5 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set094.csv
train 6 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set017.csv
train 7 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set074.csv
train 8 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set055.csv
train 9 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set096.csv
train 10 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set054.csv
train 11 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set093.csv
train 12 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set079.csv
train 13 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set014.csv
train 14 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set008.csv
train 15 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set031.csv
train 16 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set023.csv
train 17 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set025.csv
train 18 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set034.csv
train 19 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set009.csv
train 20 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set044.csv
train 21 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set063.csv
train 22 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set004.csv
train 23 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set072.csv
train 24 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set046.csv
train 25 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set049.csv
train 26 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set007.csv
train 27 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set100.csv
train 28 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set083.csv
train 29 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set077.csv
train 30 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set061.csv
train 31 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set081.csv
train 32 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set091.csv
train 33 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set069.csv
train 34 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set052.csv
train 35 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set028.csv
train 36 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set019.csv
train 37 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set057.csv
train 38 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set064.csv
train 39 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set075.csv
train 40 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set002.csv
train 41 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set062.csv
train 42 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set043.csv
train 43 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set042.csv
train 44 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set005.csv
train 45 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set016.csv
train 46 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set018.csv
train 47 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set041.csv
train 48 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set039.csv
train 49 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set006.csv
train 50 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set092.csv
train 51 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set060.csv
train 52 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set001.csv
train 53 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set035.csv
train 54 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set029.csv
train 55 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set051.csv
train 56 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set012.csv
train 57 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set036.csv
train 58 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set024.csv
train 59 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set053.csv
train 60 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set011.csv
train 61 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set032.csv
train 62 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set067.csv
train 63 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set058.csv
train 64 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set080.csv
train 65 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set086.csv
train 66 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set033.csv
train 67 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set085.csv
train 68 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set015.csv
train 69 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set090.csv
train 70 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set020.csv
train 71 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set030.csv
train 72 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set050.csv
train 73 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set098.csv
train 74 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set099.csv
train 75 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set070.csv
train 76 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set021.csv
train 77 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set095.csv
train 78 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set073.csv
train 79 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set078.csv
test 80 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set026.csv
test 81 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set038.csv
test 82 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set082.csv
test 83 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set047.csv
test 84 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set040.csv
test 85 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set066.csv
test 86 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set059.csv
test 87 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set013.csv
test 88 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set089.csv
test 89 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set071.csv
test 90 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set088.csv
test 91 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set037.csv
test 92 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set022.csv
test 93 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set084.csv
test 94 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set010.csv
test 95 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set097.csv
test 96 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set068.csv
test 97 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set065.csv
test 98 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set048.csv
test 99 /beegfs/ye53nis/saves/firstartefact_Sep2019/traces_cluster_rand_Sep2019_set045.csv
shapes of feature dataframe: (20000, 8000) and label dataframe: (20000, 8000)
shapes of feature dataframe: (20000, 2000) and label dataframe: (20000, 2000)

for each 20,000 timestap trace there are the following numbers of corrupted timesteps:
 label001_1    6286
label001_1    2568
label001_1    4495
label001_1    4414
label001_1    1105
dtype: int64
2020-07-03 13:52:37.767508: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-07-03 13:52:37.782937: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2194930000 Hz
2020-07-03 13:52:37.784247: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55e467051fe0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-07-03 13:52:37.784290: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
number of training examples: 6400, number of validation examples: 1600

------------------------
number of test examples: 2000

input - shape:   (None, 8192, 1)
output - shape:  (None, 8192, 1)
2020-07-03 13:52:43.215627: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/training/tracking/data_structures.py:739: DeprecationWarning: Using or importing the ABCs from 'collections' instead
 of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  if not isinstance(wrapped_dict, collections.Mapping):
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:60: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature()
 or inspect.getfullargspec()
  all_param_names, _, _, all_default_values = inspect.getargspec(fn)  # pylint: disable=W1505
Epoch 1/40
/home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:347: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from
'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  if not isinstance(values, collections.Sequence):
   1/1280 [..............................] - ETA: 0s - loss: 1.2862 - tp: 16168.0000 - fp: 12373.0000 - tn: 7437.0000 - fn: 4982.0000 - precision: 0.5665 - recall: 0.7644 - accuracy: 0.5763 - auc: 0.63882020-07-03 13:52:58.138783: I ten
sorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
WARNING:tensorflow:From /home/ye53nis/.conda/envs/mlflow-c61a56dd06e99e6740b7159c7af9d54a736a4e4b/lib/python3.8/site-packages/tensorflow/python/ops/summary_ops_v2.py:1277: stop (from tensorflow.python.eager.profiler) is deprecated and w
ill be removed after 2020-07-01.
Instructions for updating:
use `tf.profiler.experimental.stop` instead.
2020-07-03 13:52:59.481566: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_07_03_13_52_59
2020-07-03 13:52:59.501564: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2020_07_03_13_52_59/node011.trace.json.gz
2020-07-03 13:52:59.538369: I tensorflow/core/profiler/rpc/client/save_profile.cc:176] Creating directory: /tmp/tb/train/plugins/profile/2020_07_03_13_52_59
2020-07-03 13:52:59.538526: I tensorflow/core/profiler/rpc/client/save_profile.cc:182] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2020_07_03_13_52_59/node011.memory_profile.json.gz
2020-07-03 13:52:59.541206: I tensorflow/python/profiler/internal/profiler_wrapper.cc:111] Creating directory: /tmp/tb/train/plugins/profile/2020_07_03_13_52_59Dumped tool data for xplane.pb to /tmp/tb/train/plugins/profile/2020_07_03_1
3_52_59/node011.xplane.pb
Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2020_07_03_13_52_59/node011.overview_page.pb
Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2020_07_03_13_52_59/node011.input_pipeline.pb
Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2020_07_03_13_52_59/node011.tensorflow_stats.pb
Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2020_07_03_13_52_59/node011.kernel_stats.pb

1280/1280 [==============================] - 1780s 1s/step - loss: 1.1858 - tp: 7339984.0000 - fp: 2874113.0000 - tn: 37719448.0000 - fn: 4495237.0000 - precision: 0.7186 - recall: 0.6202 - accuracy: 0.8594 - auc: 0.8832 - val_loss: 2.1
472 - val_tp: 2536431.0000 - val_fp: 7204421.0000 - val_tn: 3012796.0000 - val_fn: 353552.0000 - val_precision: 0.2604 - val_recall: 0.8777 - val_accuracy: 0.4234 - val_auc: 0.7624
Epoch 2/40
1280/1280 [==============================] - 1779s 1s/step - loss: 1.0617 - tp: 7423177.0000 - fp: 2931677.0000 - tn: 37621800.0000 - fn: 4452150.0000 - precision: 0.7169 - recall: 0.6251 - accuracy: 0.8592 - auc: 0.8778 - val_loss: 3.9
428 - val_tp: 2825884.0000 - val_fp: 7982935.0000 - val_tn: 2238761.0000 - val_fn: 59620.0000 - val_precision: 0.2614 - val_recall: 0.9793 - val_accuracy: 0.3864 - val_auc: 0.8231
Epoch 3/40
1280/1280 [==============================] - 1777s 1s/step - loss: 0.7857 - tp: 8301792.0000 - fp: 2630361.0000 - tn: 37976972.0000 - fn: 3519672.0000 - precision: 0.7594 - recall: 0.7023 - accuracy: 0.8827 - auc: 0.9097 - val_loss: 66.
2140 - val_tp: 3034899.0000 - val_fp: 9956908.0000 - val_tn: 115041.0000 - val_fn: 352.0000 - val_precision: 0.2336 - val_recall: 0.9999 - val_accuracy: 0.2403 - val_auc: 0.5115
Epoch 4/40
1280/1280 [==============================] - 1760s 1s/step - loss: 0.6022 - tp: 9369523.0000 - fp: 2098170.0000 - tn: 38477736.0000 - fn: 2483339.0000 - precision: 0.8170 - recall: 0.7905 - accuracy: 0.9126 - auc: 0.9419 - val_loss: 1.4
866 - val_tp: 2661698.0000 - val_fp: 4816937.0000 - val_tn: 5366742.0000 - val_fn: 261823.0000 - val_precision: 0.3559 - val_recall: 0.9104 - val_accuracy: 0.6125 - val_auc: 0.8933
Epoch 5/40
1280/1280 [==============================] - 1771s 1s/step - loss: 0.5592 - tp: 9674903.0000 - fp: 1962418.0000 - tn: 38513608.0000 - fn: 2277853.0000 - precision: 0.8314 - recall: 0.8094 - accuracy: 0.9191 - auc: 0.9488 - val_loss: 1.0
996 - val_tp: 565756.0000 - val_fp: 6305.0000 - val_tn: 10145255.0000 - val_fn: 2389884.0000 - val_precision: 0.9890 - val_recall: 0.1914 - val_accuracy: 0.8172 - val_auc: 0.9230
Epoch 6/40
1280/1280 [==============================] - 1774s 1s/step - loss: 0.5210 - tp: 9808780.0000 - fp: 1924251.0000 - tn: 38545780.0000 - fn: 2149975.0000 - precision: 0.8360 - recall: 0.8202 - accuracy: 0.9223 - auc: 0.9528 - val_loss: 1.9
950 - val_tp: 2773239.0000 - val_fp: 6335919.0000 - val_tn: 3933010.0000 - val_fn: 65032.0000 - val_precision: 0.3044 - val_recall: 0.9771 - val_accuracy: 0.5116 - val_auc: 0.9026
Epoch 7/40
1280/1280 [==============================] - 1767s 1s/step - loss: 0.4799 - tp: 9927548.0000 - fp: 1743956.0000 - tn: 38779680.0000 - fn: 1977563.0000 - precision: 0.8506 - recall: 0.8339 - accuracy: 0.9290 - auc: 0.9587 - val_loss: 0.4
051 - val_tp: 2537115.0000 - val_fp: 260830.0000 - val_tn: 9808854.0000 - val_fn: 500401.0000 - val_precision: 0.9068 - val_recall: 0.8353 - val_accuracy: 0.9419 - val_auc: 0.9654
Epoch 8/40
1280/1280 [==============================] - 1771s 1s/step - loss: 0.4655 - tp: 9972310.0000 - fp: 1610651.0000 - tn: 38928064.0000 - fn: 1917779.0000 - precision: 0.8609 - recall: 0.8387 - accuracy: 0.9327 - auc: 0.9611 - val_loss: 1.6
014 - val_tp: 1146828.0000 - val_fp: 2196.0000 - val_tn: 10286639.0000 - val_fn: 1671537.0000 - val_precision: 0.9981 - val_recall: 0.4069 - val_accuracy: 0.8723 - val_auc: 0.7663
Epoch 9/40
1280/1280 [==============================] - 1770s 1s/step - loss: 0.4410 - tp: 9975626.0000 - fp: 1547724.0000 - tn: 39046088.0000 - fn: 1859375.0000 - precision: 0.8657 - recall: 0.8429 - accuracy: 0.9350 - auc: 0.9628 - val_loss: 0.6
413 - val_tp: 2584588.0000 - val_fp: 554056.0000 - val_tn: 9603651.0000 - val_fn: 364905.0000 - val_precision: 0.8235 - val_recall: 0.8763 - val_accuracy: 0.9299 - val_auc: 0.9693
Epoch 10/40
1280/1280 [==============================] - 1761s 1s/step - loss: 0.4217 - tp: 10018262.0000 - fp: 1432905.0000 - tn: 39172740.0000 - fn: 1804910.0000 - precision: 0.8749 - recall: 0.8473 - accuracy: 0.9382 - auc: 0.9649 - val_loss: 0.
6441 - val_tp: 1928738.0000 - val_fp: 27586.0000 - val_tn: 10052344.0000 - val_fn: 1098532.0000 - val_precision: 0.9859 - val_recall: 0.6371 - val_accuracy: 0.9141 - val_auc: 0.9337
Epoch 11/40
1280/1280 [==============================] - 1767s 1s/step - loss: 0.4095 - tp: 10121675.0000 - fp: 1466373.0000 - tn: 39131704.0000 - fn: 1709053.0000 - precision: 0.8735 - recall: 0.8555 - accuracy: 0.9394 - auc: 0.9669 - val_loss: 1.
0952 - val_tp: 2779103.0000 - val_fp: 2929574.0000 - val_tn: 7332225.0000 - val_fn: 66298.0000 - val_precision: 0.4868 - val_recall: 0.9767 - val_accuracy: 0.7714 - val_auc: 0.9444
Epoch 12/40
1280/1280 [==============================] - 1766s 1s/step - loss: 0.3508 - tp: 10558538.0000 - fp: 1308014.0000 - tn: 39148768.0000 - fn: 1413463.0000 - precision: 0.8898 - recall: 0.8819 - accuracy: 0.9481 - auc: 0.9727 - val_loss: 0.
4481 - val_tp: 2197978.0000 - val_fp: 77726.0000 - val_tn: 10018559.0000 - val_fn: 812937.0000 - val_precision: 0.9658 - val_recall: 0.7300 - val_accuracy: 0.9320 - val_auc: 0.9684
Epoch 13/40
1280/1280 [==============================] - 1768s 1s/step - loss: 0.3459 - tp: 10634920.0000 - fp: 1277214.0000 - tn: 39126304.0000 - fn: 1390329.0000 - precision: 0.8928 - recall: 0.8844 - accuracy: 0.9491 - auc: 0.9732 - val_loss: 0.
4022 - val_tp: 2391272.0000 - val_fp: 184993.0000 - val_tn: 9955529.0000 - val_fn: 575406.0000 - val_precision: 0.9282 - val_recall: 0.8060 - val_accuracy: 0.9420 - val_auc: 0.9744
Epoch 14/40
1280/1280 [==============================] - 1771s 1s/step - loss: 0.3308 - tp: 10495827.0000 - fp: 1232895.0000 - tn: 39364996.0000 - fn: 1335085.0000 - precision: 0.8949 - recall: 0.8872 - accuracy: 0.9510 - auc: 0.9745 - val_loss: 0.
3876 - val_tp: 2364208.0000 - val_fp: 159464.0000 - val_tn: 9940852.0000 - val_fn: 642676.0000 - val_precision: 0.9368 - val_recall: 0.7863 - val_accuracy: 0.9388 - val_auc: 0.9704
Epoch 15/40
1280/1280 [==============================] - 1778s 1s/step - loss: 0.3349 - tp: 10513378.0000 - fp: 1247874.0000 - tn: 39301756.0000 - fn: 1365808.0000 - precision: 0.8939 - recall: 0.8850 - accuracy: 0.9501 - auc: 0.9742 - val_loss: 0.
4307 - val_tp: 2108827.0000 - val_fp: 89619.0000 - val_tn: 10175358.0000 - val_fn: 733396.0000 - val_precision: 0.9592 - val_recall: 0.7420 - val_accuracy: 0.9372 - val_auc: 0.9676
Epoch 16/40
1280/1280 [==============================] - 1774s 1s/step - loss: 0.3307 - tp: 10595335.0000 - fp: 1236928.0000 - tn: 39255112.0000 - fn: 1341378.0000 - precision: 0.8955 - recall: 0.8876 - accuracy: 0.9508 - auc: 0.9744 - val_loss: 0.
4616 - val_tp: 2311103.0000 - val_fp: 115583.0000 - val_tn: 10000757.0000 - val_fn: 679757.0000 - val_precision: 0.9524 - val_recall: 0.7727 - val_accuracy: 0.9393 - val_auc: 0.9704
Epoch 17/40
1280/1280 [==============================] - 1767s 1s/step - loss: 0.3289 - tp: 10535633.0000 - fp: 1223924.0000 - tn: 39330720.0000 - fn: 1338507.0000 - precision: 0.8959 - recall: 0.8873 - accuracy: 0.9511 - auc: 0.9748 - val_loss: 0.
4047 - val_tp: 2225304.0000 - val_fp: 97248.0000 - val_tn: 10022325.0000 - val_fn: 762323.0000 - val_precision: 0.9581 - val_recall: 0.7448 - val_accuracy: 0.9344 - val_auc: 0.9659
Epoch 18/40
1280/1280 [==============================] - 1772s 1s/step - loss: 0.3248 - tp: 10613594.0000 - fp: 1201191.0000 - tn: 39285904.0000 - fn: 1328110.0000 - precision: 0.8983 - recall: 0.8888 - accuracy: 0.9518 - auc: 0.9753 - val_loss: 0.
5139 - val_tp: 2072016.0000 - val_fp: 36763.0000 - val_tn: 10080347.0000 - val_fn: 918074.0000 - val_precision: 0.9826 - val_recall: 0.6930 - val_accuracy: 0.9272 - val_auc: 0.9530
Epoch 19/40
1280/1280 [==============================] - 1774s 1s/step - loss: 0.3211 - tp: 10667086.0000 - fp: 1209967.0000 - tn: 39238632.0000 - fn: 1313139.0000 - precision: 0.8981 - recall: 0.8904 - accuracy: 0.9519 - auc: 0.9753 - val_loss: 0.
3749 - val_tp: 2493058.0000 - val_fp: 176033.0000 - val_tn: 9926603.0000 - val_fn: 511506.0000 - val_precision: 0.9340 - val_recall: 0.8298 - val_accuracy: 0.9475 - val_auc: 0.9723
Epoch 20/40
1280/1280 [==============================] - 1771s 1s/step - loss: 0.3176 - tp: 10604079.0000 - fp: 1175083.0000 - tn: 39349048.0000 - fn: 1300586.0000 - precision: 0.9002 - recall: 0.8907 - accuracy: 0.9528 - auc: 0.9758 - val_loss: 0.
4352 - val_tp: 2225431.0000 - val_fp: 169949.0000 - val_tn: 9987117.0000 - val_fn: 724703.0000 - val_precision: 0.9291 - val_recall: 0.7543 - val_accuracy: 0.9317 - val_auc: 0.9660
Epoch 21/40
1280/1280 [==============================] - 1777s 1s/step - loss: 0.3183 - tp: 10753725.0000 - fp: 1207924.0000 - tn: 39171120.0000 - fn: 1296058.0000 - precision: 0.8990 - recall: 0.8924 - accuracy: 0.9522 - auc: 0.9760 - val_loss: 0.
5143 - val_tp: 1923402.0000 - val_fp: 8402.0000 - val_tn: 10080301.0000 - val_fn: 1095095.0000 - val_precision: 0.9957 - val_recall: 0.6372 - val_accuracy: 0.9158 - val_auc: 0.9559
Epoch 22/40
1280/1280 [==============================] - 1772s 1s/step - loss: 0.3155 - tp: 10593298.0000 - fp: 1171798.0000 - tn: 39369164.0000 - fn: 1294568.0000 - precision: 0.9004 - recall: 0.8911 - accuracy: 0.9530 - auc: 0.9766 - val_loss: 0.
3698 - val_tp: 2362795.0000 - val_fp: 98123.0000 - val_tn: 10020849.0000 - val_fn: 625433.0000 - val_precision: 0.9601 - val_recall: 0.7907 - val_accuracy: 0.9448 - val_auc: 0.9722
Epoch 23/40
1280/1280 [==============================] - 1770s 1s/step - loss: 0.3081 - tp: 10733583.0000 - fp: 1175054.0000 - tn: 39255684.0000 - fn: 1264464.0000 - precision: 0.9013 - recall: 0.8946 - accuracy: 0.9535 - auc: 0.9772 - val_loss: 0.
4154 - val_tp: 2216921.0000 - val_fp: 66152.0000 - val_tn: 10061258.0000 - val_fn: 762869.0000 - val_precision: 0.9710 - val_recall: 0.7440 - val_accuracy: 0.9368 - val_auc: 0.9674
Epoch 24/40
1280/1280 [==============================] - 1773s 1s/step - loss: 0.3108 - tp: 10737070.0000 - fp: 1185865.0000 - tn: 39247804.0000 - fn: 1258035.0000 - precision: 0.9005 - recall: 0.8951 - accuracy: 0.9534 - auc: 0.9769 - val_loss: 0.
4095 - val_tp: 2189914.0000 - val_fp: 65716.0000 - val_tn: 10122627.0000 - val_fn: 728943.0000 - val_precision: 0.9709 - val_recall: 0.7503 - val_accuracy: 0.9394 - val_auc: 0.9691
Epoch 25/40
1280/1280 [==============================] - 1778s 1s/step - loss: 0.3092 - tp: 10574009.0000 - fp: 1168082.0000 - tn: 39411864.0000 - fn: 1274842.0000 - precision: 0.9005 - recall: 0.8924 - accuracy: 0.9534 - auc: 0.9769 - val_loss: 0.
3740 - val_tp: 2271238.0000 - val_fp: 99057.0000 - val_tn: 10104250.0000 - val_fn: 632655.0000 - val_precision: 0.9582 - val_recall: 0.7821 - val_accuracy: 0.9442 - val_auc: 0.9706
Epoch 26/40
1280/1280 [==============================] - 1772s 1s/step - loss: 0.3127 - tp: 10661534.0000 - fp: 1178414.0000 - tn: 39303136.0000 - fn: 1285691.0000 - precision: 0.9005 - recall: 0.8924 - accuracy: 0.9530 - auc: 0.9766 - val_loss: 0.
4100 - val_tp: 2160141.0000 - val_fp: 93000.0000 - val_tn: 10111722.0000 - val_fn: 742337.0000 - val_precision: 0.9587 - val_recall: 0.7442 - val_accuracy: 0.9363 - val_auc: 0.9656
Epoch 27/40
1280/1280 [==============================] - 1772s 1s/step - loss: 0.3054 - tp: 10663139.0000 - fp: 1174175.0000 - tn: 39348628.0000 - fn: 1242863.0000 - precision: 0.9008 - recall: 0.8956 - accuracy: 0.9539 - auc: 0.9770 - val_loss: 0.
3826 - val_tp: 2295025.0000 - val_fp: 98538.0000 - val_tn: 10045922.0000 - val_fn: 667715.0000 - val_precision: 0.9588 - val_recall: 0.7746 - val_accuracy: 0.9415 - val_auc: 0.9695
Epoch 28/40
1280/1280 [==============================] - 1766s 1s/step - loss: 0.3101 - tp: 10638913.0000 - fp: 1161985.0000 - tn: 39375784.0000 - fn: 1252135.0000 - precision: 0.9015 - recall: 0.8947 - accuracy: 0.9540 - auc: 0.9766 - val_loss: 0.
3907 - val_tp: 2166669.0000 - val_fp: 37875.0000 - val_tn: 10143155.0000 - val_fn: 759501.0000 - val_precision: 0.9828 - val_recall: 0.7404 - val_accuracy: 0.9392 - val_auc: 0.9686
Epoch 29/40
1280/1280 [==============================] - 1773s 1s/step - loss: 0.3063 - tp: 10554201.0000 - fp: 1162046.0000 - tn: 39456136.0000 - fn: 1256420.0000 - precision: 0.9008 - recall: 0.8936 - accuracy: 0.9539 - auc: 0.9767 - val_loss: 0.
3597 - val_tp: 2286152.0000 - val_fp: 91290.0000 - val_tn: 10085049.0000 - val_fn: 644709.0000 - val_precision: 0.9616 - val_recall: 0.7800 - val_accuracy: 0.9438 - val_auc: 0.9718
Epoch 30/40
1280/1280 [==============================] - 1777s 1s/step - loss: 0.3054 - tp: 10552880.0000 - fp: 1163086.0000 - tn: 39473676.0000 - fn: 1239134.0000 - precision: 0.9007 - recall: 0.8949 - accuracy: 0.9542 - auc: 0.9771 - val_loss: 0.
4182 - val_tp: 2195277.0000 - val_fp: 32841.0000 - val_tn: 10111337.0000 - val_fn: 767745.0000 - val_precision: 0.9853 - val_recall: 0.7409 - val_accuracy: 0.9389 - val_auc: 0.9704
Epoch 31/40
1280/1280 [==============================] - 1779s 1s/step - loss: 0.3083 - tp: 10553649.0000 - fp: 1149510.0000 - tn: 39466480.0000 - fn: 1259216.0000 - precision: 0.9018 - recall: 0.8934 - accuracy: 0.9541 - auc: 0.9765 - val_loss: 0.
4366 - val_tp: 2167915.0000 - val_fp: 58039.0000 - val_tn: 10066034.0000 - val_fn: 815212.0000 - val_precision: 0.9739 - val_recall: 0.7267 - val_accuracy: 0.9334 - val_auc: 0.9673
Epoch 32/40
1280/1280 [==============================] - 1768s 1s/step - loss: 0.3090 - tp: 10850631.0000 - fp: 1193472.0000 - tn: 39117152.0000 - fn: 1267532.0000 - precision: 0.9009 - recall: 0.8954 - accuracy: 0.9531 - auc: 0.9774 - val_loss: 0.
3760 - val_tp: 2399785.0000 - val_fp: 89881.0000 - val_tn: 9935425.0000 - val_fn: 682109.0000 - val_precision: 0.9639 - val_recall: 0.7787 - val_accuracy: 0.9411 - val_auc: 0.9706
Epoch 33/40
1280/1280 [==============================] - 1770s 1s/step - loss: 0.3075 - tp: 10591815.0000 - fp: 1173635.0000 - tn: 39401344.0000 - fn: 1261969.0000 - precision: 0.9002 - recall: 0.8935 - accuracy: 0.9535 - auc: 0.9770 - val_loss: 0.
3749 - val_tp: 2266851.0000 - val_fp: 113739.0000 - val_tn: 10069260.0000 - val_fn: 657350.0000 - val_precision: 0.9522 - val_recall: 0.7752 - val_accuracy: 0.9412 - val_auc: 0.9685
Epoch 34/40
1280/1280 [==============================] - 1774s 1s/step - loss: 0.3046 - tp: 10617309.0000 - fp: 1139098.0000 - tn: 39400540.0000 - fn: 1271864.0000 - precision: 0.9031 - recall: 0.8930 - accuracy: 0.9540 - auc: 0.9772 - val_loss: 0.
4175 - val_tp: 2241531.0000 - val_fp: 87208.0000 - val_tn: 10047153.0000 - val_fn: 731308.0000 - val_precision: 0.9626 - val_recall: 0.7540 - val_accuracy: 0.9376 - val_auc: 0.9662
Epoch 35/40
1280/1280 [==============================] - 1773s 1s/step - loss: 0.3112 - tp: 10604940.0000 - fp: 1177338.0000 - tn: 39382680.0000 - fn: 1263836.0000 - precision: 0.9001 - recall: 0.8935 - accuracy: 0.9534 - auc: 0.9763 - val_loss: 0.
3769 - val_tp: 2350085.0000 - val_fp: 99126.0000 - val_tn: 10001938.0000 - val_fn: 656051.0000 - val_precision: 0.9595 - val_recall: 0.7818 - val_accuracy: 0.9424 - val_auc: 0.9697
Epoch 36/40
1280/1280 [==============================] - 1782s 1s/step - loss: 0.3079 - tp: 10576699.0000 - fp: 1181888.0000 - tn: 39421312.0000 - fn: 1248879.0000 - precision: 0.8995 - recall: 0.8944 - accuracy: 0.9536 - auc: 0.9767 - val_loss: 0.
3834 - val_tp: 2322480.0000 - val_fp: 71956.0000 - val_tn: 10012406.0000 - val_fn: 700358.0000 - val_precision: 0.9699 - val_recall: 0.7683 - val_accuracy: 0.9411 - val_auc: 0.9705
Epoch 37/40
1280/1280 [==============================] - 1775s 1s/step - loss: 0.3129 - tp: 10722707.0000 - fp: 1163322.0000 - tn: 39255528.0000 - fn: 1287232.0000 - precision: 0.9021 - recall: 0.8928 - accuracy: 0.9533 - auc: 0.9765 - val_loss: 0.
4019 - val_tp: 2314261.0000 - val_fp: 62408.0000 - val_tn: 9985730.0000 - val_fn: 744801.0000 - val_precision: 0.9737 - val_recall: 0.7565 - val_accuracy: 0.9384 - val_auc: 0.9702
Epoch 38/40
1280/1280 [==============================] - 1783s 1s/step - loss: 0.3083 - tp: 10594205.0000 - fp: 1165411.0000 - tn: 39412032.0000 - fn: 1257165.0000 - precision: 0.9009 - recall: 0.8939 - accuracy: 0.9538 - auc: 0.9767 - val_loss: 0.
3835 - val_tp: 2223637.0000 - val_fp: 52165.0000 - val_tn: 10117258.0000 - val_fn: 714140.0000 - val_precision: 0.9771 - val_recall: 0.7569 - val_accuracy: 0.9415 - val_auc: 0.9715
Epoch 39/40
1280/1280 [==============================] - 1776s 1s/step - loss: 0.3073 - tp: 10690271.0000 - fp: 1156059.0000 - tn: 39323200.0000 - fn: 1259274.0000 - precision: 0.9024 - recall: 0.8946 - accuracy: 0.9539 - auc: 0.9769 - val_loss: 0.
3965 - val_tp: 2204023.0000 - val_fp: 67630.0000 - val_tn: 10139880.0000 - val_fn: 695667.0000 - val_precision: 0.9702 - val_recall: 0.7601 - val_accuracy: 0.9418 - val_auc: 0.9679
Epoch 40/40
1280/1280 [==============================] - 1771s 1s/step - loss: 0.3049 - tp: 10682842.0000 - fp: 1183309.0000 - tn: 39323412.0000 - fn: 1239227.0000 - precision: 0.9003 - recall: 0.8961 - accuracy: 0.9538 - auc: 0.9773 - val_loss: 0.
3758 - val_tp: 2309020.0000 - val_fp: 116124.0000 - val_tn: 10004803.0000 - val_fn: 677253.0000 - val_precision: 0.9521 - val_recall: 0.7732 - val_accuracy: 0.9395 - val_auc: 0.9702
400/400 [==============================] - 118s 296ms/step - loss: 0.4356 - tp: 2969562.0000 - fp: 289346.0000 - tn: 12303178.0000 - fn: 821914.0000 - precision: 0.9112 - recall: 0.7832 - accuracy: 0.9322 - auc: 0.9648
2020/07/04 09:37:42 INFO mlflow.projects: === Run (ID 'd9b44dc2e3d44ea1a71129808b642af6') succeeded ===

2.2 exp-201231-clustersim

  • this experiment is to document the simulation of fluorescence timetraces with "bright cluster" artifacts

2.2.1 connect to jupyter notebook

  1. Request compute node via tmux
cd /
srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
(tf-nightly) [ye53nis@node146 /]$ jupyter lab --no-browser --port=$PORT
[I 00:02:39.372 LabApp] JupyterLab extension loaded from /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/jupyterlab
[I 00:02:39.372 LabApp] JupyterLab application directory is /home/ye53nis/.conda/envs/tf-nightly/share/jupyter/lab
[I 00:02:39.375 LabApp] Serving notebooks from local directory: /
[I 00:02:39.375 LabApp] Jupyter Notebook 6.1.4 is running at:
[I 00:02:39.375 LabApp] http://localhost:9999/?token=93791464f12bacd92a8343c1a3be84117c0674d5703fd278
[I 00:02:39.375 LabApp]  or http://127.0.0.1:9999/?token=93791464f12bacd92a8343c1a3be84117c0674d5703fd278
[I 00:02:39.375 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 00:02:39.380 LabApp]

    To access the notebook, open this file in a browser:
        file:///home/ye53nis/.local/share/jupyter/runtime/nbserver-54410-open.html
    Or copy and paste one of these URLs:
        http://localhost:9999/?token=93791464f12bacd92a8343c1a3be84117c0674d5703fd278
     or http://127.0.0.1:9999/?token=93791464f12bacd92a8343c1a3be84117c0674d5703fd278
  1. Create SSH tunnel
sh-5.0$ sh-5.0$ ye53nis@ara-login01.rz.uni-jena.de's password:          
ye53nis@node146's password:              
Last login: Mon Jan 4 12:09:39 2021 from login01.ara
  1. connect to Python 3 kernel using jupyter-server-list-kernels
    python3           f6f8ea4a-c473-459c-93b6-4984b0987ff8   a few seconds ago    starting   0
    

2.2.2 record metadata

%cd /beegfs/ye53nis/drmed-git/
/beegfs/ye53nis/drmed-git
!git log -1
!git status
commit 90315560e472cfec38b0f927e905da1635d25240
Author: Apoplex <oligolex@vivaldi.net>
Date:   Thu Dec 31 01:32:25 2020 +0100

    change metadata gathering and add docs
# On branch exp-201231-clustersim
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#   (commit or discard the untracked or modified content in submodules)
#
#	modified:   src/nanosimpy (untracked content)
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#	data/
#	experiment_params.csv
#	mlruns/
#	tramp.YDPCnB
no changes added to commit (use "git add" and/or "git commit -a")

#+CALL: jp-metadata(_long='True)

No of CPUs in system: 72
No of CPUs the current process can use: 24
load average: (16.0, 16.04, 16.01)
os.uname():  posix.uname_result(sysname='Linux', nodename='node146', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
PID of process: 173820
RAM total: 199G, RAM used: 70G, RAM free: 99G
the current directory: /beegfs/ye53nis/drmed-git
My disk usage:
Filesystem           Size  Used Avail Use% Mounted on
/dev/sda1             50G  3.2G   47G   7% /
devtmpfs              94G     0   94G   0% /dev
tmpfs                 94G  297M   94G   1% /dev/shm
tmpfs                 94G  195M   94G   1% /run
tmpfs                 94G     0   94G   0% /sys/fs/cgroup
nfs01-ib:/cluster    2.0T  473G  1.6T  24% /cluster
nfs03-ib:/pool/work  100T   70T   31T  70% /nfsdata
nfs02-ib:/data01      88T   71T   17T  81% /data01
nfs01-ib:/home        80T   71T  9.8T  88% /home
/dev/sda3            6.0G  435M  5.6G   8% /var
/dev/sda5            2.0G   34M  2.0G   2% /tmp
/dev/sda6            169G   18G  152G  11% /local
beegfs_nodev         524T  437T   88T  84% /beegfs
tmpfs                 19G     0   19G   0% /run/user/67339
# packages in environment at /home/ye53nis/.conda/envs/tf-nightly:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
absl-py                   0.11.0                   pypi_0    pypi
alembic                   1.4.1                      py_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
argon2-cffi               20.1.0           py38h7b6447c_1
asn1crypto                1.4.0              pyh9f0ad1d_0    conda-forge
asteval                   0.9.16             pyh5ca1d4c_0    conda-forge
astunparse                1.6.3                    pypi_0    pypi
async_generator           1.10                       py_0
attrs                     20.2.0                     py_0
azure-core                1.8.2              pyh9f0ad1d_0    conda-forge
azure-storage-blob        12.5.0             pyh9f0ad1d_0    conda-forge
backcall                  0.2.0                      py_0
blas                      1.0                         mkl
bleach                    3.2.1                      py_0
blinker                   1.4                        py_1    conda-forge
brotlipy                  0.7.0           py38h7b6447c_1000
ca-certificates           2020.12.5            ha878542_0    conda-forge
cachetools                4.1.1                    pypi_0    pypi
certifi                   2020.12.5        py38h578d9bd_0    conda-forge
cffi                      1.14.3           py38he30daa8_0
chardet                   3.0.4                 py38_1003
click                     7.1.2              pyh9f0ad1d_0    conda-forge
cloudpickle               1.6.0                      py_0    conda-forge
configparser              5.0.1                      py_0    conda-forge
cryptography              3.1.1            py38h1ba5d50_0
cycler                    0.10.0                   py38_0
databricks-cli            0.9.1                      py_0    conda-forge
dbus                      1.13.18              hb2f20db_0
decorator                 4.4.2                      py_0
defusedxml                0.6.0                      py_0
docker-py                 4.3.1            py38h32f6830_1    conda-forge
docker-pycreds            0.4.0                      py_0    conda-forge
entrypoints               0.3                      py38_0
expat                     2.2.10               he6710b0_2
fcsfiles                  2020.9.18                pypi_0    pypi
flask                     1.1.2              pyh9f0ad1d_0    conda-forge
flatbuffers               1.12                     pypi_0    pypi
fontconfig                2.13.0               h9420a91_0
freetype                  2.10.4               h5ab3b9f_0
future                    0.18.2           py38h578d9bd_2    conda-forge
gast                      0.3.3                    pypi_0    pypi
gitdb                     4.0.5                      py_0    conda-forge
gitpython                 3.1.11                     py_0    conda-forge
glib                      2.66.1               h92f7085_0
google-auth               1.23.0                   pypi_0    pypi
google-auth-oauthlib      0.4.2                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
gorilla                   0.3.0                      py_0    conda-forge
grpcio                    1.32.0                   pypi_0    pypi
gst-plugins-base          1.14.0               hbbd80ab_1
gstreamer                 1.14.0               hb31296c_0
gunicorn                  20.0.4           py38h32f6830_2    conda-forge
h5py                      2.10.0                   pypi_0    pypi
icu                       58.2                 he6710b0_3
idna                      2.10                       py_0
importlib-metadata        2.0.0                      py_1
importlib_metadata        2.0.0                         1
intel-openmp              2020.2                      254
ipykernel                 5.3.4            py38h5ca1d4c_0
ipython                   7.18.1           py38h5ca1d4c_0
ipython_genutils          0.2.0                    py38_0
isodate                   0.6.0                      py_1    conda-forge
itsdangerous              1.1.0                      py_0    conda-forge
jedi                      0.17.2                   py38_0
jinja2                    2.11.2                     py_0
jpeg                      9b                   h024ee3a_2
json5                     0.9.5                      py_0
jsonschema                3.2.0                      py_2
jupyter_client            6.1.7                      py_0
jupyter_core              4.6.3                    py38_0
jupyterlab                2.2.6                      py_0
jupyterlab_pygments       0.1.2                      py_0
jupyterlab_server         1.2.0                      py_0
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.3.0            py38h2531618_0
lcms2                     2.11                 h396b838_0
ld_impl_linux-64          2.33.1               h53a641e_7
libedit                   3.1.20191231         h14c3975_1
libffi                    3.3                  he6710b0_2
libgcc-ng                 9.1.0                hdf63c60_0
libgfortran-ng            7.3.0                hdf63c60_0
libpng                    1.6.37               hbc83047_0
libprotobuf               3.13.0.1             h8b12597_0    conda-forge
libsodium                 1.0.18               h7b6447c_0
libstdcxx-ng              9.1.0                hdf63c60_0
libtiff                   4.1.0                h2733197_1
libuuid                   1.0.3                h1bed415_2
libxcb                    1.14                 h7b6447c_0
libxml2                   2.9.10               hb55368b_3
lmfit                     1.0.1                      py_1    conda-forge
lz4-c                     1.9.2                heb0550a_3
mako                      1.1.3              pyh9f0ad1d_0    conda-forge
markdown                  3.3.3                    pypi_0    pypi
markupsafe                1.1.1            py38h7b6447c_0
matplotlib                3.3.2                         0
matplotlib-base           3.3.2            py38h817c723_0
mistune                   0.8.4           py38h7b6447c_1000
mkl                       2020.2                      256
mkl-service               2.3.0            py38he904b0f_0
mkl_fft                   1.2.0            py38h23d657b_0
mkl_random                1.1.1            py38h0573a6f_0
mlflow                    1.11.0           py38h32f6830_1    conda-forge
msrest                    0.6.19             pyh9f0ad1d_0    conda-forge
multipletau               0.3.3                    pypi_0    pypi
nbclient                  0.5.1                      py_0
nbconvert                 6.0.7                    py38_0
nbformat                  5.0.8                      py_0
ncurses                   6.2                  he6710b0_1
nest-asyncio              1.4.1                      py_0
notebook                  6.1.4                    py38_0
numpy                     1.19.2           py38h54aff64_0
numpy-base                1.19.2           py38hfa32c7d_0
oauthlib                  3.0.1                      py_0    conda-forge
olefile                   0.46                       py_0
openssl                   1.1.1h               h516909a_0    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 20.4                       py_0
pandas                    1.1.3            py38he6710b0_0
pandoc                    2.11                 hb0f4dca_0
pandocfilters             1.4.2                    py38_1
parso                     0.7.0                      py_0
pcre                      8.44                 he6710b0_0
pexpect                   4.8.0                    py38_0
pickleshare               0.7.5                 py38_1000
pillow                    8.0.1            py38he98fc37_0
pip                       20.2.4                   py38_0
prometheus_client         0.8.0                      py_0
prometheus_flask_exporter 0.18.1             pyh9f0ad1d_0    conda-forge
prompt-toolkit            3.0.8                      py_0
protobuf                  3.13.0.1         py38h950e882_1    conda-forge
ptyprocess                0.6.0                    py38_0
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycparser                 2.20                       py_2
pygments                  2.7.2              pyhd3eb1b0_0
pyjwt                     1.7.1                      py_0    conda-forge
pyopenssl                 19.1.0                     py_1
pyparsing                 2.4.7                      py_0
pyqt                      5.9.2            py38h05f1152_4
pyrsistent                0.17.3           py38h7b6447c_0
pysocks                   1.7.1                    py38_0
python                    3.8.5                h7579374_1
python-dateutil           2.8.1                      py_0
python-editor             1.0.4                      py_0    conda-forge
python_abi                3.8                      1_cp38    conda-forge
pytz                      2020.1                     py_0
pyyaml                    5.3.1            py38h8df0ef7_1    conda-forge
pyzmq                     19.0.2           py38he6710b0_1
qt                        5.9.7                h5867ecd_1
querystring_parser        1.2.4                      py_0    conda-forge
readline                  8.0                  h7b6447c_0
requests                  2.24.0                     py_0
requests-oauthlib         1.3.0              pyh9f0ad1d_0    conda-forge
rsa                       4.6                      pypi_0    pypi
scipy                     1.5.2            py38h0b6359f_0
seaborn                   0.11.0                     py_0
send2trash                1.5.0                    py38_0
setuptools                50.3.0           py38hb0f4dca_1
sip                       4.19.13          py38he6710b0_0
six                       1.15.0                     py_0
smmap                     3.0.4              pyh9f0ad1d_0    conda-forge
sqlalchemy                1.3.13           py38h516909a_0    conda-forge
sqlite                    3.33.0               h62c20be_0
sqlparse                  0.4.1              pyh9f0ad1d_0    conda-forge
tabulate                  0.8.7              pyh9f0ad1d_0    conda-forge
tb-nightly                2.4.0a20201102           pypi_0    pypi
tensorboard-plugin-wit    1.7.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
terminado                 0.9.1                    py38_0
testpath                  0.4.4                      py_0
tf-estimator-nightly      2.4.0.dev2020102301          pypi_0    pypi
tf-nightly                2.5.0.dev20201029          pypi_0    pypi
tifffile                  2020.10.1        py38hdd07704_2
tk                        8.6.10               hbc83047_0
tornado                   6.0.4            py38h7b6447c_1
traitlets                 5.0.5                      py_0
typing-extensions         3.7.4.3                  pypi_0    pypi
uncertainties             3.1.5              pyhd8ed1ab_0    conda-forge
urllib3                   1.25.11                    py_0
wcwidth                   0.2.5                      py_0
webencodings              0.5.1                    py38_1
websocket-client          0.57.0           py38h32f6830_3    conda-forge
werkzeug                  1.0.1              pyh9f0ad1d_0    conda-forge
wheel                     0.35.1                     py_0
wrapt                     1.12.1                   pypi_0    pypi
xz                        5.2.5                h7b6447c_0
yaml                      0.2.5                h516909a_0    conda-forge
zeromq                    4.3.3                he6710b0_3
zipp                      3.4.0              pyhd3eb1b0_0
zlib                      1.2.11               h7b6447c_3
zstd                      1.4.5                h9ceee32_0

Note: you may need to restart the kernel to use updated packages.
{'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
 'SLURM_NODELIST': 'node146',
 'SLURM_JOB_NAME': 'bash',
 'XDG_SESSION_ID': '9639',
 'SLURMD_NODENAME': 'node146',
 'SLURM_TOPOLOGY_ADDR': 'node146',
 'SLURM_NTASKS_PER_NODE': '24',
 'HOSTNAME': 'login01',
 'SLURM_PRIO_PROCESS': '0',
 'SLURM_SRUN_COMM_PORT': '43120',
 'SHELL': '/bin/bash',
 'TERM': 'xterm-color',
 'SLURM_JOB_QOS': 'qstand',
 'SLURM_PTY_WIN_ROW': '24',
 'HISTSIZE': '1000',
 'TMPDIR': '/tmp',
 'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
 'SSH_CLIENT': '10.231.210.198 43508 22',
 'CONDA_SHLVL': '2',
 'CONDA_PROMPT_MODIFIER': '(tf-nightly) ',
 'WINDOWID': '0',
 'QTDIR': '/usr/lib64/qt-3.3',
 'QTINC': '/usr/lib64/qt-3.3/include',
 'SSH_TTY': '/dev/pts/5',
 'QT_GRAPHICSSYSTEM_CHECKED': '1',
 'SLURM_NNODES': '1',
 'USER': 'ye53nis',
 'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
 'CONDA_EXE': '/cluster/miniconda3/bin/conda',
 'SLURM_STEP_NUM_NODES': '1',
 'SLURM_JOBID': '534856',
 'SRUN_DEBUG': '3',
 'SLURM_NTASKS': '24',
 'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
 'SLURM_STEP_ID': '0',
 'TMUX': '/tmp/tmux-67339/default,27827,6',
 '_CE_CONDA': '',
 'CONDA_PREFIX_1': '/cluster/miniconda3',
 'SLURM_STEP_LAUNCHER_PORT': '43120',
 'SLURM_TASKS_PER_NODE': '24',
 'MAIL': '/var/spool/mail/ye53nis',
 'PATH': '/home/ye53nis/.conda/envs/tf-nightly/bin:/home/lex/Programme/miniconda3/envs/tf-nightly-lab/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/home/lex/Programme/miniconda3/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
 'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
 'SLURM_JOB_ID': '534856',
 'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tf-nightly',
 'SLURM_JOB_USER': 'ye53nis',
 'SLURM_STEPID': '0',
 'PWD': '/',
 'SLURM_SRUN_COMM_HOST': '192.168.192.5',
 'LANG': 'en_US.UTF-8',
 'SLURM_PTY_WIN_COL': '80',
 'SLURM_UMASK': '0022',
 'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
 'SLURM_JOB_UID': '67339',
 'LOADEDMODULES': '',
 'SLURM_NODEID': '0',
 'TMUX_PANE': '%6',
 'SLURM_SUBMIT_DIR': '/',
 'SLURM_TASK_PID': '53476',
 'SLURM_NPROCS': '24',
 'SLURM_CPUS_ON_NODE': '24',
 'SLURM_DISTRIBUTION': 'block',
 'https_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
 'SLURM_PROCID': '0',
 'HISTCONTROL': 'ignoredups',
 '_CE_M': '',
 'SLURM_JOB_NODELIST': 'node146',
 'SLURM_PTY_PORT': '46638',
 'HOME': '/home/ye53nis',
 'SHLVL': '3',
 'SLURM_LOCALID': '0',
 'SLURM_JOB_GID': '13280',
 'SLURM_JOB_CPUS_PER_NODE': '24',
 'SLURM_CLUSTER_NAME': 'hpc',
 'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
 'SLURM_SUBMIT_HOST': 'login01',
 'SLURM_JOB_PARTITION': 's_standard',
 'MATHEMATICA_HOME': '/cluster/apps/mathematica/11.3',
 'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
 'LOGNAME': 'ye53nis',
 'SLURM_STEP_NUM_TASKS': '24',
 'QTLIB': '/usr/lib64/qt-3.3/lib',
 'SLURM_JOB_ACCOUNT': 'iaob',
 'SLURM_JOB_NUM_NODES': '1',
 'MODULESHOME': '/usr/share/Modules',
 'CONDA_DEFAULT_ENV': 'tf-nightly',
 'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
 'SLURM_STEP_TASKS_PER_NODE': '24',
 'PORT': '9999',
 'SLURM_STEP_NODELIST': 'node146',
 'DISPLAY': ':0',
 'XDG_RUNTIME_DIR': '',
 'XAUTHORITY': '/home/lex/.Xauthority',
 'BASH_FUNC_module()': '() {  eval `/usr/bin/modulecmd bash $*`\n}',
 '_': '/home/ye53nis/.conda/envs/tf-nightly/bin/jupyter',
 'JPY_PARENT_PID': '54410',
 'CLICOLOR': '1',
 'PAGER': 'cat',
 'GIT_PAGER': 'cat',
 'MPLBACKEND': 'module://ipykernel.pylab.backend_inline'}

2.2.3 set variables

  • any files generated using the :file header of org-mode source blocks will be saved here (Note: the destination of the simulations is different, see the variables below)
(setq org-babel-jupyter-resource-directory "./data/exp-201231-clustsim")
./data/exp-201231-clustsim
import sys
sys.path.append('/beegfs/ye53nis/drmed-git/src/')
from fluotracify.simulations import simulate_trace_with_artifact as stwa
folder = '/beegfs/ye53nis/saves/firstartifact_Nov2020/'
file_name = 'traces_brightclust_Nov2020'
total_sim_time = 16384
d_mol_arr = [0.069, 0.08, 0.1, 0.2, 0.4, 0.6, 1.0, 3.0, 10, 50]
col_per_example = 3
label_for = 'both'
number_of_sets = 10
traces_per_set = 100
artifact = 1
  • for each diffusion constant given in d_mol_arr, 10 .csv files will be generated (number_of_sets) which each comprise of 100 fluorescence traces (traces_per_set)
  • The output of the function will be written to stdout (the terminal inside a tmux session on the machine where the notebook is running). Thus, Emacs can be closed. After the simulations are done, the printed output can be copied from the terminal, because tmux is making it possible to attach and detach to a running terminal session.
sys.stdout = open('/dev/stdout', 'w')

2.2.4 Do the simulation

stwa.produce_training_data(folder=folder,
                           file_name=file_name,
                           col_per_example=col_per_example,
                           number_of_sets=number_of_sets,
                           traces_per_set=traces_per_set,
                           total_sim_time=total_sim_time,
                           artifact=1,
                           d_mol_arr=d_mol_arr,
                           label_for=label_for)
354a0d72-5d50-4e51-b971-a52c0cf8f572
  • Note: the results were printed to the terminal, but I didn't think of the terminal history limit of tmux, which is at around 1800 lines. This means, most of the history couldn't be saved. Here is an example of the output:
    Set 10 ------------------------
    num_of_steps 16384
    Processing tracks: [=================== ] 99% complete
    Processing FWHM 250, num_of_steps 16384
    Processing tracks: [=================   ] 85% complete
    Processing FWHM 250,
    Trace 1: Nmol: 2621 d_mol: 50 Cluster multiplier: 9000
    num_of_steps 16384
    Processing tracks: [=================== ] 99% complete
    Processing FWHM 250, num_of_steps 16384
    Processing tracks: [=================   ] 85% complete
    Processing FWHM 250,
    Trace 2: Nmol: 2621 d_mol: 50 Cluster multiplier: 5000
    num_of_steps 16384
    Processing tracks: [=================== ] 99% complete
    Processing FWHM 250, num_of_steps 16384
    Processing tracks: [=================   ] 85% complete
    Processing FWHM 250,
    Trace 3: Nmol: 2621 d_mol: 50 Cluster multiplier: 5000
    num_of_steps 16384
    Processing tracks: [=================== ] 99% complete
    Processing FWHM 250, num_of_steps 16384
    Processing tracks: [=================   ] 85% complete
    Processing FWHM 250,
    Trace 4: Nmol: 2621 d_mol: 50 Cluster multiplier: 9000
    num_of_steps 16384
    Processing tracks: [=================== ] 99% complete
    Processing FWHM 250, num_of_steps 16384
    Processing tracks: [=================   ] 85% complete
    Processing FWHM 250,
    Trace 5: Nmol: 2621 d_mol: 50 Cluster multiplier: 7000
    num_of_steps 16384
    Processing tracks: [=================== ] 99% complete
    Processing FWHM 250, num_of_steps 16384
    Processing tracks: [=================   ] 85% complete
    Processing FWHM 250,
    Trace 6: Nmol: 2621 d_mol: 50 Cluster multiplier: 8000
    num_of_steps 16384
    Processing tracks: [=================== ] 99% complete
    Processing FWHM 250, num_of_steps 16384
    Processing tracks: [=================   ] 85% complete
    Processing FWHM 250,
    Trace 7: Nmol: 2621 d_mol: 50 Cluster multiplier: 7000
    num_of_steps 16384
    Processing tracks: [=================== ] 99% complete
    Processing FWHM 250, num_of_steps 16384
    Processing tracks: [=================   ] 85% complete
    Processing FWHM 250,
    Trace 8: Nmol: 2621 d_mol: 50 Cluster multiplier: 5000
    num_of_steps 16384
    Processing tracks: [=================== ] 99% complete
    Processing FWHM 250, num_of_steps 16384
    Processing tracks: [=================   ] 85% complete
    Processing FWHM 250,
    Trace 9: Nmol: 2621 d_mol: 50 Cluster multiplier: 9000
    num_of_steps 16384
    Processing tracks: [=================== ] 99% complete
    Processing FWHM 250, num_of_steps 16384
    Processing tracks: [=================   ] 85% complete
    Processing FWHM 250,
    Trace 10: Nmol: 2621 d_mol: 50 Cluster multiplier: 9000
    num_of_steps 16384
    Processing tracks: [=================== ] 99% complete
    Processing FWHM 250, num_of_steps 16384
    Processing tracks: [=================   ] 85% complete
    Processing FWHM 250,
    

2.2.5 preparing examplary plots of the results (new kernel, new code)

  • I used a different jupyter kernel for loading (See properties drawer above), thus libraries have to be imported again and parameters have to be set again. On the plus side: the sequence of plotting here is independent of the sequence of simulation above.
    %cd /beegfs/ye53nis/drmed-git/
    
    /beegfs/ye53nis/drmed-git
    
  • I also used some additional code for plotting:
    !git log -1
    
    !git status
    
    commit 47d02edd7313d1b172934a17a0aca0a8e47a8fff
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Tue Jan 5 21:41:11 2021 +0100
    
        Add docs; rename to drate to diffrate
    # On branch exp-201231-clustersim
    # Changes not staged for commit:
    #   (use "git add <file>..." to update what will be committed)
    #   (use "git checkout -- <file>..." to discard changes in working directory)
    #   (commit or discard the untracked or modified content in submodules)
    #
    #	modified:   src/nanosimpy (untracked content)
    #
    # Untracked files:
    #   (use "git add <file>..." to include in what will be committed)
    #
    #	data/
    #	experiment_params.csv
    #	mlruns/
    #	test.pdf
    #	test.svg
    #	tramp.YDPCnB
    no changes added to commit (use "git add" and/or "git commit -a")
    
  • again the file to save plots in - notice that this is on my local machine, not the remote one, since I open this LabBook.org-file on my local machine.
    (setq org-babel-jupyter-resource-directory "./data/exp-201231-clustsim")
    
    ./data/exp-201231-clustsim
    
  • then the necessary imports and variables
    import sys
    sys.path.append('/beegfs/ye53nis/drmed-git/src/')
    from fluotracify.simulations import import_simulation_from_csv as isfc
    from fluotracify.simulations import plot_simulations as ps
    
    folder = '/beegfs/ye53nis/saves/firstartifact_Nov2020/'
    col_per_example = 3
    d_mol_arr = [0.069, 0.08, 0.1, 0.2, 0.4, 0.6, 1.0, 3.0, 10, 50]
    artifact = 1
    
  • let's load our data
    dataset, _, nsamples, experiment_params = isfc.import_from_csv(
        folder=folder,
        header=12,
        frac_train=1,
        col_per_example=col_per_example,
        dropindex=None,
        dropcolumns=None)
    
    train 0 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set007.csv
    train 1 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set007.csv
    train 2 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set003.csv
    train 3 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set006.csv
    train 4 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set006.csv
    train 5 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set004.csv
    train 6 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set007.csv
    train 7 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set004.csv
    train 8 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set005.csv
    train 9 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set006.csvtrain 10 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set004.csv
    train 11 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set003.csv
    train 12 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set009.csv
    train 13 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set004.csv
    train 14 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set008.csv
    train 15 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set001.csv
    train 16 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set003.csv
    train 17 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set005.csv
    train 18 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set004.csv
    train 19 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set009.csv
    train 20 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set004.csv
    train 21 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set003.csv
    train 22 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set004.csv
    train 23 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set002.csv
    train 24 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set006.csv
    train 25 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set009.csv
    train 26 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set007.csv
    train 27 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set010.csv
    train 28 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set003.csv
    train 29 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set007.csv
    train 30 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
    train 31 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set001.csv
    train 32 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set001.csv
    train 33 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set009.csv
    train 34 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set002.csv
    train 35 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set008.csv
    train 36 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set009.csv
    train 37 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set007.csv
    train 38 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set004.csv
    train 39 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set005.csv
    train 40 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set002.csv
    train 41 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
    train 42 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set003.csv
    train 43 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set002.csv
    train 44 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set005.csv
    train 45 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set006.csv
    train 46 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set008.csv
    train 47 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set001.csv
    train 48 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set009.csv
    train 49 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set006.csv
    train 50 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set002.csv
    train 51 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set010.csv
    train 52 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set001.csv
    train 53 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set005.csv
    train 54 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set009.csv
    train 55 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set001.csv
    train 56 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set002.csv
    train 57 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set006.csv
    train 58 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set004.csv
    train 59 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set003.csv
    train 60 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set001.csv
    train 61 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set002.csv
    train 62 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set007.csv
    train 63 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set008.csv
    train 64 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set010.csv
    train 65 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set006.csv
    train 66 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set003.csv
    train 67 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set005.csv
    train 68 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set005.csv
    train 69 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set010.csv
    train 70 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set010.csv
    train 71 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set010.csv
    train 72 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set010.csv
    train 73 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set008.csv
    train 74 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set009.csv
    train 75 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set010.csv
    train 76 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set001.csv
    train 77 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set005.csv
    train 78 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set003.csv
    train 79 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set008.csv
    train 80 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set006.csv
    train 81 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set008.csv
    train 82 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set002.csv
    train 83 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set007.csv
    train 84 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set010.csv
    train 85 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set006.csv
    train 86 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set009.csv
    train 87 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set003.csv
    train 88 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set009.csv
    train 89 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set001.csv
    train 90 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set008.csv
    train 91 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set007.csv
    train 92 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set002.csv
    train 93 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set004.csv
    train 94 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set010.csv
    train 95 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set007.csv
    train 96 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
    train 97 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set005.csv
    train 98 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set008.csv
    train 99 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set005.csv
    
    out = isfc.separate_data_and_labels(
        array=dataset,
        nsamples=nsamples,
        col_per_example=col_per_example)
    
    The given DataFrame was split into 3 parts with shapes: [(16384, 10000), (16384, 10000), (16384, 10000)]
    
  • added <2022-08-22 Mo>: For machine learning, we did 3 splits of data: train, validation, test. Let's load them and characterize them
    %cd /beegfs/ye53nis/drmed-git
    
    /beegfs/ye53nis/drmed-git
    
    !git log -1
    
    !git status
    
    commit 6def0df6b174f2f51efc279d6add74e9f809e1eb
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Fri Jul 15 14:36:15 2022 +0200
    
        fix correlate_timetrace_and_save
    # On branch exp-220227-unet
    # Changes not staged for commit:
    #   (use "git add <file>..." to update what will be committed)
    #   (use "git checkout -- <file>..." to discard changes in working directory)
    #   (commit or discard the untracked or modified content in submodules)
    #
    #	modified:   src/nanosimpy (untracked content)
    #
    # Untracked files:
    #   (use "git add <file>..." to include in what will be committed)
    #
    #	data/0.069.svg
    #	data/exp-210204-unet/
    #	data/exp-220120-correlate-ptu/
    #	data/exp-220227-unet/2022-03-01_experimental/
    #	data/exp-220227-unet/2022-04-22_simulations/
    #	data/exp-220227-unet/2022-05-17_simulations/
    #	data/exp-220227-unet/2022-05-22_experimental/
    #	data/exp-220227-unet/2022-06-02_experimental-pex5/
    #	data/exp-220316-publication1/
    #	data/exp-devtest/
    #	data/exp-test/
    #	data/mlruns/
    #	data/tb/
    #	experiment_params.csv
    #	src/fluotracify/applications/correlate_cython.c
    #	test
    #	tramp.YDPCnB
    no changes added to commit (use "git add" and/or "git commit -a")
    
    import sys
    
    import numpy as np
    import pandas as pd
    
    
    from pathlib import Path
    from pprint import pprint
    
    FLUOTRACIFY_PATH = '/beegfs/ye53nis/drmed-git/src/'
    sys.path.append(FLUOTRACIFY_PATH)
    from fluotracify.simulations import (
       import_simulation_from_csv as isfc,
       analyze_simulations as ans,
    )
    
    
    2022-08-22 14:53:59.575866: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2022-08-22 14:53:59.575902: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    
    col_per_example = 3
    lab_thresh = 0.04
    artifact = 0
    model_type = 1
    fwhm = 250
    
    train_path = Path('/beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets')
    val_path = Path('/beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN')
    test_path = Path('/beegfs/ye53nis/saves/firstartifact_Nov2020_test')
    
    train, _, nsamples_train, train_params = isfc.import_from_csv(
        folder=train_path,
        header=12,
        frac_train=1,
        col_per_example=col_per_example,
        dropindex=None,
        dropcolumns=None)
    
    val, _, nsamples_val, val_params = isfc.import_from_csv(
        folder=val_path,
        header=12,
        frac_train=1,
        col_per_example=col_per_example,
        dropindex=None,
        dropcolumns=None)
    
    test, _, nsamples_test, test_params = isfc.import_from_csv(
        folder=test_path,
        header=12,
        frac_train=1,
        col_per_example=col_per_example,
        dropindex=None,
        dropcolumns=None)
    
    
    2022-08-22 15:01:07,583 - sim import tools - 1/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set004.csv
    2022-08-22 15:01:10,900 - sim import tools - 2/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set002.csv
    2022-08-22 15:01:14,230 - sim import tools - 3/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set001.csv
    2022-08-22 15:01:24,427 - sim import tools - 4/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set007.csv
    2022-08-22 15:01:28,527 - sim import tools - 5/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
    2022-08-22 15:01:32,186 - sim import tools - 6/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/1.0/traces_brightclust_Nov2020_D10_set010.csv
    2022-08-22 15:01:36,667 - sim import tools - 7/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set004.csv
    2022-08-22 15:01:40,508 - sim import tools - 8/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.1/traces_brightclust_Nov2020_D3.0_set010.csv
    2022-08-22 15:01:44,080 - sim import tools - 9/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set009.csv
    2022-08-22 15:01:49,585 - sim import tools - 10/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set003.csv2022-08-22 15:01:53,212 - sim import tools - 11/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set007.csv
    2022-08-22 15:01:56,689 - sim import tools - 12/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
    2022-08-22 15:02:00,568 - sim import tools - 13/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set003.csv
    2022-08-22 15:02:04,222 - sim import tools - 14/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set002.csv
    2022-08-22 15:02:07,872 - sim import tools - 15/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set005.csv
    2022-08-22 15:02:11,754 - sim import tools - 16/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/1.0/traces_brightclust_Nov2020_D0.069_set009.csv
    2022-08-22 15:02:16,110 - sim import tools - 17/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set004.csv
    2022-08-22 15:02:19,635 - sim import tools - 18/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set010.csv
    2022-08-22 15:02:23,317 - sim import tools - 19/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set009.csv
    2022-08-22 15:02:26,787 - sim import tools - 20/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.01/traces_brightclust_Nov2020_D1.0_set010.csv
    2022-08-22 15:02:30,350 - sim import tools - 21/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set007.csv
    2022-08-22 15:02:38,204 - sim import tools - 22/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set010.csv
    2022-08-22 15:02:42,002 - sim import tools - 23/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set001.csv
    2022-08-22 15:02:47,861 - sim import tools - 24/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set003.csv
    2022-08-22 15:02:51,915 - sim import tools - 25/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set003.csv
    2022-08-22 15:02:55,479 - sim import tools - 26/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set004.csv
    2022-08-22 15:02:59,060 - sim import tools - 27/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set001.csv
    2022-08-22 15:03:02,729 - sim import tools - 28/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set005.csv
    2022-08-22 15:03:06,402 - sim import tools - 29/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set006.csv
    2022-08-22 15:03:10,838 - sim import tools - 30/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set004.csv
    2022-08-22 15:03:15,520 - sim import tools - 31/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set006.csv
    2022-08-22 15:03:19,243 - sim import tools - 32/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set004.csv
    2022-08-22 15:03:24,959 - sim import tools - 33/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set005.csv
    2022-08-22 15:03:28,685 - sim import tools - 34/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set003.csv
    2022-08-22 15:03:32,498 - sim import tools - 35/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.01/traces_brightclust_Nov2020_D0.2_set003.csv
    2022-08-22 15:03:36,044 - sim import tools - 36/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.01/traces_brightclust_Nov2020_D50_set006.csv
    2022-08-22 15:03:39,848 - sim import tools - 37/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set004.csv
    2022-08-22 15:03:43,605 - sim import tools - 38/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set004.csv
    2022-08-22 15:03:47,474 - sim import tools - 39/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.01/traces_brightclust_Nov2020_D0.6_set010.csv
    2022-08-22 15:03:51,072 - sim import tools - 40/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set007.csv
    2022-08-22 15:03:54,988 - sim import tools - 41/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set006.csv
    2022-08-22 15:04:01,497 - sim import tools - 42/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set002.csv
    2022-08-22 15:04:05,120 - sim import tools - 43/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set006.csv
    2022-08-22 15:04:08,664 - sim import tools - 44/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set006.csv
    2022-08-22 15:04:13,252 - sim import tools - 45/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set010.csv
    2022-08-22 15:04:16,976 - sim import tools - 46/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.01/traces_brightclust_Nov2020_D0.08_set007.csv
    2022-08-22 15:04:20,744 - sim import tools - 47/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set002.csv
    2022-08-22 15:04:24,635 - sim import tools - 48/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set009.csv
    2022-08-22 15:04:28,258 - sim import tools - 1/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/0.1/traces_brightclust_Nov2020_D1.0_set009.csv
    2022-08-22 15:04:31,883 - sim import tools - 2/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/50/1.0/traces_brightclust_Nov2020_D50_set007.csv
    2022-08-22 15:04:36,746 - sim import tools - 3/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.4/0.1/traces_brightclust_Nov2020_D0.4_set009.csv
    2022-08-22 15:04:40,324 - sim import tools - 4/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/1.0/traces_brightclust_Nov2020_D3.0_set009.csv
    2022-08-22 15:04:44,167 - sim import tools - 5/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.1/0.01/traces_brightclust_Nov2020_D0.1_set008.csv
    2022-08-22 15:04:47,733 - sim import tools - 6/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/10/0.01/traces_brightclust_Nov2020_D10_set008.csv
    2022-08-22 15:04:51,261 - sim import tools - 7/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.08/0.1/traces_brightclust_Nov2020_D0.08_set008.csv
    2022-08-22 15:04:54,837 - sim import tools - 8/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
    2022-08-22 15:04:58,638 - sim import tools - 9/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/0.01/traces_brightclust_Nov2020_D3.0_set008.csv
    2022-08-22 15:05:02,079 - sim import tools - 10/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.2/0.1/traces_brightclust_Nov2020_D0.2_set006.csv
    2022-08-22 15:05:06,190 - sim import tools - 11/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.069/0.1/traces_brightclust_Nov2020_D0.069_set006.csv
    2022-08-22 15:05:09,643 - sim import tools - 12/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.6/0.1/traces_brightclust_Nov2020_D0.6_set006.csv
    2022-08-22 15:05:13,347 - sim import tools - 1/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.069/1.0/traces_brightclust_Nov2020_D0.069_set010.csv
    2022-08-22 15:05:17,852 - sim import tools - 2/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/50/0.1/traces_brightclust_Nov2020_D50_set003.csv
    2022-08-22 15:05:21,482 - sim import tools - 3/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.4/0.1/traces_brightclust_Nov2020_D0.4_set001.csv
    2022-08-22 15:05:25,196 - sim import tools - 4/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.2/0.1/traces_brightclust_Nov2020_D0.2_set007.csv
    2022-08-22 15:05:29,300 - sim import tools - 5/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/3.0/1.0/traces_brightclust_Nov2020_D3.0_set002.csv
    2022-08-22 15:05:32,904 - sim import tools - 6/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/3.0/0.01/traces_brightclust_Nov2020_D3.0_set004.csv
    2022-08-22 15:05:36,427 - sim import tools - 7/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/50/0.01/traces_brightclust_Nov2020_D50_set002.csv
    2022-08-22 15:05:42,126 - sim import tools - 8/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.2/1.0/traces_brightclust_Nov2020_D0.2_set005.csv
    2022-08-22 15:05:45,767 - sim import tools - 9/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.6/1.0/traces_brightclust_Nov2020_D0.6_set009.csv
    2022-08-22 15:05:49,081 - sim import tools - 10/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/10/0.1/traces_brightclust_Nov2020_D10_set001.csv
    2022-08-22 15:05:52,579 - sim import tools - 11/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.08/1.0/traces_brightclust_Nov2020_D0.08_set001.csv
    2022-08-22 15:05:56,075 - sim import tools - 12/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.6/0.1/traces_brightclust_Nov2020_D0.6_set003.csv
    2022-08-22 15:05:59,546 - sim import tools - 13/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.1/1.0/traces_brightclust_Nov2020_D0.1_set001.csv
    2022-08-22 15:06:03,365 - sim import tools - 14/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.4/1.0/traces_brightclust_Nov2020_D0.4_set005.csv
    2022-08-22 15:06:06,954 - sim import tools - 15/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/10/1.0/traces_brightclust_Nov2020_D10_set005.csv
    2022-08-22 15:06:10,482 - sim import tools - 16/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/1.0/1.0/traces_brightclust_Nov2020_D1.0_set005.csv
    2022-08-22 15:06:15,475 - sim import tools - 17/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.069/0.1/traces_brightclust_Nov2020_D0.069_set001.csv
    2022-08-22 15:06:19,122 - sim import tools - 18/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/50/1.0/traces_brightclust_Nov2020_D50_set001.csv
    2022-08-22 15:06:22,559 - sim import tools - 19/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.1/0.01/traces_brightclust_Nov2020_D0.1_set002.csv
    2022-08-22 15:06:26,128 - sim import tools - 20/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.08/0.1/traces_brightclust_Nov2020_D0.08_set003.csv
    2022-08-22 15:06:29,503 - sim import tools - 21/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/1.0/0.01/traces_brightclust_Nov2020_D1.0_set006.csv
    2022-08-22 15:06:32,862 - sim import tools - 22/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/1.0/0.1/traces_brightclust_Nov2020_D1.0_set003.csv
    2022-08-22 15:06:36,597 - sim import tools - 23/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.2/0.01/traces_brightclust_Nov2020_D0.2_set002.csv
    2022-08-22 15:06:40,322 - sim import tools - 24/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.1/0.1/traces_brightclust_Nov2020_D0.1_set005.csv
    2022-08-22 15:06:43,806 - sim import tools - 25/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/3.0/0.1/traces_brightclust_Nov2020_D3.0_set007.csv
    2022-08-22 15:06:47,273 - sim import tools - 26/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.08/0.01/traces_brightclust_Nov2020_D0.08_set005.csv
    2022-08-22 15:06:51,150 - sim import tools - 27/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.069/0.01/traces_brightclust_Nov2020_D0.069_set005.csv
    2022-08-22 15:06:54,884 - sim import tools - 28/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/10/0.01/traces_brightclust_Nov2020_D10_set002.csv
    2022-08-22 15:06:58,405 - sim import tools - 29/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.6/0.01/traces_brightclust_Nov2020_D0.6_set008.csv
    2022-08-22 15:07:01,776 - sim import tools - 30/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.4/0.01/traces_brightclust_Nov2020_D0.4_set008.csv
    
    train_sep = isfc.separate_data_and_labels(array=train,
                                            nsamples=nsamples_train,
                                            col_per_example=col_per_example)
    
    val_sep = isfc.separate_data_and_labels(array=val,
                                            nsamples=nsamples_val,
                                            col_per_example=col_per_example)
    
    test_sep = isfc.separate_data_and_labels(array=test,
                                            nsamples=nsamples_test,
                                            col_per_example=col_per_example)
    
    train_dirty = train_sep['0']
    train_labels = train_sep['1']
    train_labbool = train_labels > lab_thresh
    train_clean = train_sep['2']
    
    val_dirty = val_sep['0']
    val_labels = val_sep['1']
    val_labbool = val_labels > lab_thresh
    val_clean = val_sep['2']
    
    test_dirty = test_sep['0']
    test_labels = test_sep['1']
    test_labbool = test_labels > lab_thresh
    test_clean = test_sep['2']
    
    train_dirty
    
    2022-08-22 15:09:46,295 - sim import tools - The given DataFrame was split into 3 parts with shapes: [(16384, 4800), (16384, 4800), (16384, 4800)]
    2022-08-22 15:09:46,389 - sim import tools - The given DataFrame was split into 3 parts with shapes: [(16384, 1200), (16384, 1200), (16384, 1200)]
    2022-08-22 15:09:46,630 - sim import tools - The given DataFrame was split into 3 parts with shapes: [(16384, 3000), (16384, 3000), (16384, 3000)]
    
      trace001 trace002 trace003 trace004 trace005 trace006 trace007 trace008 trace009 trace010 trace091 trace092 trace093 trace094 trace095 trace096 trace097 trace098 trace099 trace100
    0 2087.936279 1859.954834 1258.620728 2111.855713 1154.932617 1681.850952 1902.115112 1953.226929 1493.605713 1824.782837 2991.938965 3056.693848 3065.942139 3209.311768 3072.823975 2137.467773 2536.288086 2596.720459 2322.556152 3241.336670
    1 2225.077148 1538.099731 1234.391113 2202.052979 1106.553467 1599.424927 1756.183716 2155.242676 1453.128052 1733.415771 2830.736816 2632.801514 2681.733887 3103.006836 2852.432373 2902.637939 2606.319336 2728.558350 2388.868164 2557.717041
    2 2283.867920 1672.847534 1297.318359 2068.382568 1088.629150 1539.257568 1730.095825 1944.324951 1615.796753 1677.323486 3424.374268 2697.639648 2094.186279 2632.571777 2324.150879 2309.326904 2808.739746 3119.658447 2337.465332 2208.561035
    3 2101.477295 1826.694214 1343.309448 1992.692871 1082.352295 1493.574097 1632.583740 2063.340576 1768.437134 1518.847534 3126.366455 2493.093018 1937.462158 2888.637939 2180.181396 2906.921631 2272.959717 2823.713379 2872.672607 3321.868408
    4 1987.587769 1914.085205 1294.796875 2188.687256 1047.618774 1740.084961 1642.488403 1969.975830 1540.356079 1479.204956 4045.698975 2669.068115 2655.093018 2531.362305 3005.655273 2545.799316 2692.775635 3238.976562 2815.497314 2685.596680
    16379 2248.828369 1512.159180 2574.389893 2085.525146 5731.513672 1902.472778 1578.385742 1916.452393 1320.047119 1550.951904 2291.134277 4845.781250 2534.955322 2397.597900 2810.859375 3058.239746 3012.957520 2055.962646 2597.934814 2275.868896
    16380 2194.083984 1545.642700 2921.095459 2010.420410 5787.336426 1907.732056 1580.937744 2172.060059 1421.129517 1479.062744 2997.078125 5184.318359 2522.363281 2385.093506 2436.397705 3615.403076 2555.819092 1970.641113 3601.866455 2630.073730
    16381 2067.856201 1610.466431 3070.954102 2197.771729 5789.680664 1971.624023 1475.542847 2084.938232 1400.463867 1298.757202 2942.168457 5496.346680 3230.807373 3597.340332 2541.044678 2924.228516 1627.710083 2388.055176 2722.604248 2373.275879
    16382 2067.812988 1689.094238 3292.006836 2390.899170 6230.407715 1930.711670 1451.472778 2083.847168 1289.194824 1380.650146 2794.364746 5706.733887 2287.560303 3041.490723 2482.794189 3858.045654 2195.243896 2333.582764 2728.599365 2198.640869
    16383 2240.335205 1743.910645 3327.093018 2357.319092 5435.189453 1860.456299 1333.890869 2038.945801 1358.094727 1312.259766 2627.845703 5518.034668 2491.838867 2519.055176 3319.664062 3653.108398 3334.266846 2495.438721 2713.831055 2330.221436

    16384 rows × 4800 columns

    train_ntot = train_labbool.size
    train_npos = train_labbool.sum().sum()
    train_nneg = train_ntot - train_npos
    train_perc = train_npos / train_ntot
    
    val_ntot = val_labbool.size
    val_npos = val_labbool.sum().sum()
    val_nneg = val_ntot - val_npos
    val_perc = val_npos / val_ntot
    
    test_ntot = test_labbool.size
    test_npos = test_labbool.sum().sum()
    test_nneg = test_ntot - test_npos
    test_perc = test_npos / test_ntot
    
    print(f'Total N train: {train_ntot}, N pos train: {train_npos}, N neg train: {train_nneg}, Percent of pos: {train_perc:.2%}')
    print(f'Total N val: {val_ntot}, N pos val: {val_npos}, N neg val: {val_nneg}, Percent of pos: {val_perc:.2%}')
    print(f'Total N test: {test_ntot}, N pos test: {test_npos}, N neg test: {test_nneg}, Percent of pos: {test_perc:.2%}')
    
    
    Total N train: 78643200, N pos train: 11969767, N neg train: 66673433, Percent of pos: 15.22%
    Total N val: 19660800, N pos val: 3162854, N neg val: 16497946, Percent of pos: 16.09%
    Total N test: 49152000, N pos test: 7556233, N neg test: 41595767, Percent of pos: 15.37%
    

2.2.6 plots of the simulated traces by diffusion rate

  • This is the plotting code:
    import matplotlib.pyplot as plt
    import numpy as np
    import pandas as pd
    
    def plot_traces_by_diffrates(ntraces, col_per_example, diffrate_of_interest,
                                 data_label_array, experiment_params, artifact):
        """A plot to examine simulated traces via
           fluotracify.simulations.simulate_trace_with_artifacts
    
        Parameters
        ----------
        ntraces : int
            The number of traces you want to plot. It determines the size of the
            plot as well, where columns are fixed at 6 and depending on ntraces
            and col_per_example the number of rows is determined.
        col_per_example : int
            Number of columns per example, first column being a trace, and then
            one or multiple labels
        diffrate_of_interest : float
            diffusion rate used to simulate the traces of interest
        data_label_array : dict of pandas DataFrames
            Contains one key per column in each simulated example. E.g. if the
            simulated features comes with two labels, the key '0' will be the
            array with the features, '1' will be the array with label A and
            '2' will be the array with label B.
        experiment_params : pandas DataFrame
            Contains metadata of the files
        artifact : {0, 1, 2, 3}
            0 = no artifact, 1 = bright clusters, 2 = detector dropout,
            3 = photobleaching
    
        Returns
        -------
        Plot of fluorescence traces and labels from their simulations
        """
        drates = experiment_params.loc[
            'diffusion rate of molecules in micrometer^2 / s']
        # get indices of diffusion rates of interest
        dindices = drates.index.where(drates == str(diffrate_of_interest))
        dindices = dindices.dropna().astype(int)
        # get indices of first of each of the 100 traces per file as an example
        tindices = dindices * 100
        if artifact == 1:
            nclusts = experiment_params.loc['number of slow clusters'][dindices]
            dclusts = experiment_params.loc[
                'diffusion rate of clusters in micrometer^2 / s'][dindices]
    
        cols = 6
        rows = int(ntraces // (cols / col_per_example) +
                   (ntraces % (cols / col_per_example) > 0))
        # share y axis only if col_per_example is 1
        sharey = col_per_example == 1
    
        fig, ax = plt.subplots(rows,
                               cols,
                               figsize=(cols * 4, rows * 4),
                               sharex=True,
                               sharey=sharey)
        fig.add_subplot(111, frameon=False)
        # hide tick and tick label of the big axes
        plt.tick_params(labelcolor='none',
                        top=False,
                        bottom=False,
                        left=False,
                        right=False)
        plt.grid(False)
        plt.xlabel("time steps in $ms$", fontsize=20)
        plt.ylabel('fluorescence intensity in $a.u.$', labelpad=20, fontsize=20)
        suptitle_height = 1 - (fig.get_figheight() * 0.004)
        plt.suptitle(t='Simulated Fluorescence Traces With D = {} '
                     '$\\frac{{\mu m^2}}{{s}}$'.format(diffrate_of_interest),
                     y=suptitle_height,
                     fontsize=20)
        traceid = 0
        for idx in range(rows):
            for jdx in range(0, cols, col_per_example):
                # first plot the trace
                try:
                    ax[idx,
                       jdx].plot(data_label_array['0'].iloc[:, tindices[traceid]])
                except IndexError:
                    break
                if artifact == 1:
                    ax[idx, jdx].set_title('trace {} ({} clusters, $D_c$ = '
                                           '{} $\\frac{{\mu m^2}}{{s}}$)'.format(
                                               traceid + 1, nclusts.iloc[traceid],
                                               dclusts.iloc[traceid]))
                else:
                    ax[idx, jdx].set_title('trace {}'.format(traceid + 1))
                ax[idx, jdx].set_ylim(0, 12000)
                for kdx in range(1, col_per_example):
                    # then plot the labels, if they are given
                    ax[idx, jdx + kdx].plot(
                        data_label_array['{}'.format(kdx)].iloc[:,
                                                                tindices[traceid]])
                    ax[idx, jdx + kdx].set_title('label {}, type {}'.format(
                        traceid + 1, kdx))
                traceid += 1
        plt.show()
    
    
  • the actual plotting function:
    for drate in d_mol_arr:
        ps.plot_traces_by_diffrates(
            ntraces=10,
            col_per_example=col_per_example,
            diffrate_of_interest=drate,
            data_label_array=out,
            experiment_params=experiment_params,
            artifact=1)
    

    50: 4683d391bdba74b0c9460a5e968d1135995e4a81.png 10: 963615ec0ac613d53e4c4b11d9b6811c6e38a7f1.png 3: f2667c8e27f421f2a47c7f307598b32a6cde5c5f.png 1: 6a89c85b30e41cbb67c477d10e7cc2239e02f081.png 0.6: 74f0fbc639df415b140d785e8bf3aa23213c31a4.png 0.4: c5d18053e9befd64e8bfc71d4e5e260b2f5606e7.png 0.2: ea43e8d0a7d8075efe27429c44e0ea99df1221ac.png 0.1: 727100134d52d8d8967e1cbb2efda097171f06f0.png 0.08: 89250a71f8711657f2b8138e2323802a1edea8af.png 0.069: 17e9c4bfb65d91e8100cd9131cabbdb1f35e62e1.png

  • open the plots with C-c C-o or toggle inline display with C-c C-x C-v or view them by themselves in the folder

2.2.7 Learnings

  • next time: save output of simulation function not to terminal, but to .txt file.

2.3 exp-210204-unet

2.3.1 Connect

2.3.1.1 Node for script execution
  1. Setup Tmux #+CALL: setup-tmux[:session local2]()
    rm: cannot remove 'home/lex.tmux-local-socket-remote-machine': No such file or directory
    ye53nis@ara-login01.rz.uni-jena.de's password:              
    /tmp/tmux-67339/default                
    > ye53nis@ara-login01.rz.uni-jena.de's password:            
  2. Request a compute node from Ara cluster
    cd /
    srun -p b_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
    
    (base) [ye53nis@node117 /]$
    
  3. Make sure you are on the desired branch (exp-210104-unet)
    cd /beegfs/ye53nis/drmed-git
    git status
    
    (base) [ye53nis@node221 /]$ cd /beegfs/ye53nis/drmed-git
    (base) [ye53nis@node221 drmed-git]$ git status
    # On branch exp-210104-unet
    no changes added to commit (use "git add" and/or "git commit -a")
    (base) [ye53nis@node221 drmed-git]$
    
2.3.1.2 Node for running Jupyter
  1. Setup Tmux (if we haven't done it before)
         
    sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:
    > ye53nis@ara-login01.rz.uni-jena.de's password:
  2. Request compute node from Ara cluster
    cd /
    srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
    
  3. Start Jupyter Lab
    (base) [ye53nis@login01 /]$ srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
    (base) [ye53nis@node302 /]$ conda activate tf-nightly
    (tf-nightly) [ye53nis@node302 /]$ export PORT=9999
    (tf-nightly) [ye53nis@node302 /]$ export XDG_RUNTIME_DIR=''
    (tf-nightly) [ye53nis@node302 /]$ export XDG_RUNTIME_DIR=""
    (tf-nightly) [ye53nis@node302 /]$ jupyter lab --no-browser --port=$PORT
    [I 21:13:32.358 LabApp] JupyterLab extension loaded from /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/jupyterlab
    [I 21:13:32.358 LabApp] JupyterLab application directory is /home/ye53nis/.conda/envs/tf-nightly/share/jupyter/lab
    [I 21:13:32.382 LabApp] Serving notebooks from local directory: /
    [I 21:13:32.382 LabApp] Jupyter Notebook 6.2.0 is running at:
    [I 21:13:32.383 LabApp] http://localhost:9999/?token=6f337c6b506799f0172382fc6a042886ebe3b00cd2a487e4
    [I 21:13:32.383 LabApp]  or http://127.0.0.1:9999/?token=6f337c6b506799f0172382fc6a042886ebe3b00cd2a487e4
    [I 21:13:32.383 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
    [C 21:13:32.418 LabApp]
    
        To access the notebook, open this file in a browser:
            file:///home/ye53nis/.local/share/jupyter/runtime/nbserver-124120-open.html
        Or copy and paste one of these URLs:
            http://localhost:9999/?token=6f337c6b506799f0172382fc6a042886ebe3b00cd2a487e4
         or http://127.0.0.1:9999/?token=6f337c6b506799f0172382fc6a042886ebe3b00cd2a487e4
    
  4. Tunnel the remote jupyter to the local computer
                     
    sh-5.1$ sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:          
    ye53nis@node302's password:              
    Last login: Fri Apr 2 19:25:33 2021 from login01.ara
  5. Start Python 3 kernel using jupyter-server-list-kernels. Then copy the kernel ID to the :PROPERTIES: drawer for any subtree where you want to use it
    python3           f26207a6-d326-45bb-b432-63d05a573ade   in a few seconds     starting   0
    
  6. any files generated using the :file header of org-mode source blocks will be saved here
    (setq org-babel-jupyter-resource-directory "./data/exp-210204-unet")
    
    ./data/exp-210204-unet
    
2.3.1.3 Node for running Mlflow UI
conda activate tf-nightly
mlflow ui --backend-store-uri file:///beegfs/ye53nis/drmed-git/data/mlruns
(tf-nightly) [ye53nis@login01 data]$ mlflow ui --backend-store-uri file:///beegfs/ye53nis/drmed-git/data/mlruns
[2021-02-08 11:55:44 +0100] [19566] [INFO] Starting gunicorn 20.0.4
[2021-02-08 11:55:44 +0100] [19566] [INFO] Listening at: http://127.0.0.1:5000 (19566)
[2021-02-08 11:55:44 +0100] [19566] [INFO] Using worker: sync
[2021-02-08 11:55:44 +0100] [19572] [INFO] Booting worker with pid: 19572
                 
sh-5.1$ sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:          
ye53nis@login01's password:              
bind: Address already in use        
Last login: Fri Apr 2 22:54:20 2021 from 10.231.191.246

2.3.2 Run 1 - full dataset

2.3.2.1 Record metadata
  1. current directory
    %cd /beegfs/ye53nis/drmed-git
    
    /beegfs/ye53nis/drmed-git
    
  2. git log
    !git log -3
    
    commit bc590b22329ea0dad425a040bb23370b3e9de3d4
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Thu Feb 4 20:40:13 2021 +0100
    
        Increase readability in html export
    
    commit bb78ded17807e0ad259440f4ae4624d1576cbba5
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Mon Feb 1 21:44:15 2021 +0100
    
        Add changes from unet branch like unet prepro
    commit 06186e40f8d4d1aa786e9268651160e96ead2223
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Mon Feb 1 21:16:42 2021 +0100
    
        Add these updates to ptu_utils from unet code
    
  3. Metadata of environment #+CALL: jp-metadata(_long='True)
    No of CPUs in system: 72
    No of CPUs the current process can use: 24
    load average: (16.0, 16.0, 16.05)
    os.uname():  posix.uname_result(sysname='Linux', nodename='node221', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
    PID of process: 231630
    RAM total: 199G, RAM used: 21G, RAM free: 138G
    the current directory: /beegfs/ye53nis/drmed-git
    My disk usage:
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/sda1             50G  3.2G   47G   7% /
    devtmpfs              94G     0   94G   0% /dev
    tmpfs                 94G  302M   94G   1% /dev/shm
    tmpfs                 94G  131M   94G   1% /run
    tmpfs                 94G     0   94G   0% /sys/fs/cgroup
    nfs03-ib:/pool/work  100T   78T   23T  78% /nfsdata
    nfs01-ib:/cluster    2.0T  412G  1.6T  21% /cluster
    nfs01-ib:/home        80T   64T   17T  80% /home
    nfs02-ib:/data01      88T   70T   19T  80% /data01
    /dev/sda6            169G   33M  169G   1% /local
    /dev/sda5            2.0G   34M  2.0G   2% /tmp
    /dev/sda3            6.0G  295M  5.8G   5% /var
    beegfs_nodev         524T  277T  248T  53% /beegfs
    tmpfs                 19G     0   19G   0% /run/user/67339# packages in environment at /home/ye53nis/.conda/envs/tf-nightly:
    #
    # Name                    Version                   Build  Channel
    _libgcc_mutex             0.1                        main
    absl-py                   0.11.0                   pypi_0    pypi
    alembic                   1.4.1                      py_0    conda-forge/label/main
    appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge/label/main
    argon2-cffi               20.1.0           py38h7b6447c_1
    asn1crypto                1.4.0              pyh9f0ad1d_0    conda-forge/label/main
    asteval                   0.9.16             pyh5ca1d4c_0    conda-forge/label/main
    astunparse                1.6.3                    pypi_0    pypi
    async_generator           1.10               pyhd3eb1b0_0
    attrs                     20.3.0             pyhd3eb1b0_0
    azure-core                1.10.0             pyhd8ed1ab_0    conda-forge/label/main
    azure-storage-blob        12.7.1             pyh44b312d_0    conda-forge/label/main
    backcall                  0.2.0              pyhd3eb1b0_0
    blas                      1.0                         mkl
    bleach                    3.2.3              pyhd3eb1b0_0
    blinker                   1.4                        py_1    conda-forge/label/main
    blosc                     1.20.1               hd408876_0
    brotli                    1.0.9                he6710b0_2
    brotlipy                  0.7.0           py38h27cfd23_1003
    brunsli                   0.1                  h2531618_0
    bzip2                     1.0.8                h7b6447c_0
    ca-certificates           2020.12.5            ha878542_0    conda-forge/label/main
    cachetools                4.2.1                    pypi_0    pypi
    certifi                   2020.12.5        py38h578d9bd_1    conda-forge/label/main
    cffi                      1.14.4           py38h261ae71_0
    chardet                   4.0.0           py38h06a4308_1003
    charls                    2.1.0                he6710b0_2
    click                     7.1.2              pyh9f0ad1d_0    conda-forge/label/main
    cloudpickle               1.6.0                      py_0    conda-forge/label/main
    configparser              5.0.1                      py_0    conda-forge/label/main
    cryptography              3.3.1            py38h3c74f83_0
    cycler                    0.10.0                   py38_0
    databricks-cli            0.9.1                      py_0    conda-forge/label/main
    dbus                      1.13.18              hb2f20db_0
    decorator                 4.4.2              pyhd3eb1b0_0
    defusedxml                0.6.0                      py_0
    docker-py                 4.4.1            py38h578d9bd_1    conda-forge/label/main
    docker-pycreds            0.4.0                      py_0    conda-forge/label/main
    entrypoints               0.3                      py38_0
    expat                     2.2.10               he6710b0_2
    fcsfiles                  2020.9.18                pypi_0    pypi
    flask                     1.1.2              pyh9f0ad1d_0    conda-forge/label/main
    flatbuffers               1.12                     pypi_0    pypi
    fontconfig                2.13.0               h9420a91_0
    freetype                  2.10.4               h5ab3b9f_0
    future                    0.18.2           py38h578d9bd_3    conda-forge/label/main
    gast                      0.4.0                    pypi_0    pypi
    giflib                    5.1.4                h14c3975_1
    gitdb                     4.0.5                      py_0    conda-forge/label/main
    gitpython                 3.1.12             pyhd8ed1ab_0    conda-forge/label/main
    glib                      2.66.1               h92f7085_0
    google-auth               1.24.0                   pypi_0    pypi
    google-auth-oauthlib      0.4.2                    pypi_0    pypi
    google-pasta              0.2.0                    pypi_0    pypi
    gorilla                   0.3.0                      py_0    conda-forge/label/main
    grpcio                    1.34.1                   pypi_0    pypi
    gst-plugins-base          1.14.0               h8213a91_2
    gstreamer                 1.14.0               h28cd5cc_2
    gunicorn                  20.0.4           py38h578d9bd_3    conda-forge/label/main
    h5py                      3.1.0                    pypi_0    pypi
    icu                       58.2                 he6710b0_3
    idna                      2.10               pyhd3eb1b0_0
    imagecodecs               2021.1.11        py38h581e88b_1
    importlib-metadata        2.0.0                      py_1
    importlib_metadata        2.0.0                         1
    intel-openmp              2020.2                      254
    ipykernel                 5.3.4            py38h5ca1d4c_0
    ipython                   7.19.0           py38hb070fc8_1
    ipython_genutils          0.2.0              pyhd3eb1b0_1
    isodate                   0.6.0                      py_1    conda-forge/label/main
    itsdangerous              1.1.0                      py_0    conda-forge/label/main
    jedi                      0.17.0                   py38_0
    jinja2                    2.11.2             pyhd3eb1b0_0
    jpeg                      9b                   h024ee3a_2
    json5                     0.9.5                      py_0
    jsonschema                3.2.0                      py_2
    jupyter_client            6.1.7                      py_0
    jupyter_core              4.7.0            py38h06a4308_0
    jupyterlab                2.2.6                      py_0
    jupyterlab_pygments       0.1.2                      py_0
    jupyterlab_server         1.2.0                      py_0
    jxrlib                    1.1                  h7b6447c_2
    keras-preprocessing       1.1.2                    pypi_0    pypi
    kiwisolver                1.3.0            py38h2531618_0
    lcms2                     2.11                 h396b838_0
    ld_impl_linux-64          2.33.1               h53a641e_7
    lerc                      2.2.1                h2531618_0
    libaec                    1.0.4                he6710b0_1
    libdeflate                1.7                  h27cfd23_5
    libedit                   3.1.20191231         h14c3975_1
    libffi                    3.3                  he6710b0_2
    libgcc-ng                 9.1.0                hdf63c60_0
    libgfortran-ng            7.3.0                hdf63c60_0
    libpng                    1.6.37               hbc83047_0
    libprotobuf               3.13.0.1             h8b12597_0    conda-forge/label/main
    libsodium                 1.0.18               h7b6447c_0
    libstdcxx-ng              9.1.0                hdf63c60_0
    libtiff                   4.1.0                h2733197_1
    libuuid                   1.0.3                h1bed415_2
    libwebp                   1.0.1                h8e7db2f_0
    libxcb                    1.14                 h7b6447c_0
    libxml2                   2.9.10               hb55368b_3
    libzopfli                 1.0.3                he6710b0_0
    lmfit                     1.0.1                      py_1    conda-forge/label/main
    lz4-c                     1.9.3                h2531618_0
    mako                      1.1.4              pyh44b312d_0    conda-forge/label/main
    markdown                  3.3.3                    pypi_0    pypi
    markupsafe                1.1.1            py38h7b6447c_0
    matplotlib                3.3.2                h06a4308_0
    matplotlib-base           3.3.2            py38h817c723_0
    mistune                   0.8.4           py38h7b6447c_1000
    mkl                       2020.2                      256
    mkl-service               2.3.0            py38he904b0f_0
    mkl_fft                   1.2.0            py38h23d657b_0
    mkl_random                1.1.1            py38h0573a6f_0
    mlflow                    1.13.1           py38h578d9bd_2    conda-forge/label/main
    msrest                    0.6.21             pyh44b312d_0    conda-forge/label/main
    multipletau               0.3.3                    pypi_0    pypi
    nbclient                  0.5.1                      py_0
    nbconvert                 6.0.7                    py38_0
    nbformat                  5.1.2              pyhd3eb1b0_1
    ncurses                   6.2                  he6710b0_1
    nest-asyncio              1.4.3              pyhd3eb1b0_0
    notebook                  6.2.0            py38h06a4308_0
    numpy                     1.19.2           py38h54aff64_0
    numpy-base                1.19.2           py38hfa32c7d_0
    oauthlib                  3.0.1                      py_0    conda-forge/label/main
    olefile                   0.46                       py_0
    openjpeg                  2.3.0                h05c96fa_1
    openssl                   1.1.1i               h27cfd23_0
    opt-einsum                3.3.0                    pypi_0    pypi
    packaging                 20.9               pyhd3eb1b0_0
    pandas                    1.2.1            py38ha9443f7_0
    pandoc                    2.11                 hb0f4dca_0
    pandocfilters             1.4.3            py38h06a4308_1
    parso                     0.8.1              pyhd3eb1b0_0
    pcre                      8.44                 he6710b0_0
    pexpect                   4.8.0              pyhd3eb1b0_3
    pickleshare               0.7.5           pyhd3eb1b0_1003
    pillow                    8.1.0            py38he98fc37_0
    pip                       20.3.3           py38h06a4308_0
    prometheus_client         0.9.0              pyhd3eb1b0_0
    prometheus_flask_exporter 0.18.1             pyh9f0ad1d_0    conda-forge/label/main
    prompt-toolkit            3.0.8                      py_0
    protobuf                  3.13.0.1         py38hadf7658_1    conda-forge/label/main
    ptyprocess                0.7.0              pyhd3eb1b0_2
    pyasn1                    0.4.8                    pypi_0    pypi
    pyasn1-modules            0.2.8                    pypi_0    pypi
    pycparser                 2.20                       py_2
    pygments                  2.7.4              pyhd3eb1b0_0
    pyjwt                     2.0.1              pyhd8ed1ab_0    conda-forge/label/main
    pyopenssl                 20.0.1             pyhd3eb1b0_1
    pyparsing                 2.4.7              pyhd3eb1b0_0
    pyqt                      5.9.2            py38h05f1152_4
    pyrsistent                0.17.3           py38h7b6447c_0
    pysocks                   1.7.1            py38h06a4308_0
    python                    3.8.5                h7579374_1
    python-dateutil           2.8.1              pyhd3eb1b0_0
    python-editor             1.0.4                      py_0    conda-forge/label/main
    python_abi                3.8                      1_cp38    conda-forge/label/main
    pytz                      2020.5             pyhd3eb1b0_0
    pyyaml                    5.3.1            py38h8df0ef7_1    conda-forge/label/main
    pyzmq                     20.0.0           py38h2531618_1
    qt                        5.9.7                h5867ecd_1
    querystring_parser        1.2.4                      py_0    conda-forge/label/main
    readline                  8.1                  h27cfd23_0
    requests                  2.25.1             pyhd3eb1b0_0
    requests-oauthlib         1.3.0              pyh9f0ad1d_0    conda-forge/label/main
    rsa                       4.7                      pypi_0    pypi
    scipy                     1.5.2            py38h0b6359f_0
    seaborn                   0.11.1             pyhd3eb1b0_0
    send2trash                1.5.0              pyhd3eb1b0_1
    setuptools                52.0.0           py38h06a4308_0
    sip                       4.19.13          py38he6710b0_0
    six                       1.15.0           py38h06a4308_0
    smmap                     4.0.0              pyh44b312d_0    conda-forge/label/main
    snappy                    1.1.8                he6710b0_0
    sqlalchemy                1.3.20           py38h1e0a361_0    conda-forge/label/main
    sqlite                    3.33.0               h62c20be_0
    sqlparse                  0.4.1              pyh9f0ad1d_0    conda-forge/label/main
    tabulate                  0.8.7              pyh9f0ad1d_0    conda-forge/label/main
    tb-nightly                2.5.0a20210130           pypi_0    pypi
    tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
    termcolor                 1.1.0                    pypi_0    pypi
    terminado                 0.9.2            py38h06a4308_0
    testpath                  0.4.4              pyhd3eb1b0_0
    tf-estimator-nightly      2.5.0.dev2021020101          pypi_0    pypi
    tf-nightly                2.5.0.dev20210130          pypi_0    pypi
    tifffile                  2021.1.14          pyhd3eb1b0_1
    tk                        8.6.10               hbc83047_0
    tornado                   6.1              py38h27cfd23_0
    traitlets                 5.0.5              pyhd3eb1b0_0
    typing-extensions         3.7.4.3                  pypi_0    pypi
    uncertainties             3.1.5              pyhd8ed1ab_0    conda-forge/label/main
    urllib3                   1.26.3             pyhd3eb1b0_0
    wcwidth                   0.2.5                      py_0
    webencodings              0.5.1                    py38_1
    websocket-client          0.57.0           py38h578d9bd_4    conda-forge/label/main
    werkzeug                  1.0.1              pyh9f0ad1d_0    conda-forge/label/main
    wheel                     0.36.2             pyhd3eb1b0_0
    wrapt                     1.12.1                   pypi_0    pypi
    xz                        5.2.5                h7b6447c_0
    yaml                      0.2.5                h516909a_0    conda-forge/label/main
    zeromq                    4.3.3                he6710b0_3
    zfp                       0.5.5                h2531618_4
    zipp                      3.4.0              pyhd3eb1b0_0
    zlib                      1.2.11               h7b6447c_3
    zstd                      1.4.5                h9ceee32_0
    
    Note: you may need to restart the kernel to use updated packages.
    {'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
     'SLURM_NODELIST': 'node221',
     'SLURM_JOB_NAME': 'bash',
     'XDG_SESSION_ID': '595',
     'SLURMD_NODENAME': 'node221',
     'SLURM_TOPOLOGY_ADDR': 'node221',
     'SLURM_NTASKS_PER_NODE': '24',
     'HOSTNAME': 'login01',
     'SLURM_PRIO_PROCESS': '0',
     'SLURM_SRUN_COMM_PORT': '33255',
     'SHELL': '/bin/bash',
     'TERM': 'xterm-color',
     'SLURM_JOB_QOS': 'qstand',
     'SLURM_PTY_WIN_ROW': '24',
     'HISTSIZE': '1000',
     'TMPDIR': '/tmp',
     'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
     'SSH_CLIENT': '10.231.182.213 44428 22',
     'CONDA_SHLVL': '2',
     'CONDA_PROMPT_MODIFIER': '(tf-nightly) ',
     'QTDIR': '/usr/lib64/qt-3.3',
     'QTINC': '/usr/lib64/qt-3.3/include',
     'SSH_TTY': '/dev/pts/2',
     'NO_PROXY': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24',
     'QT_GRAPHICSSYSTEM_CHECKED': '1',
     'SLURM_NNODES': '1',
     'USER': 'ye53nis',
     'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
     'CONDA_EXE': '/cluster/miniconda3/bin/conda',
     'SLURM_STEP_NUM_NODES': '1',
     'SLURM_JOBID': '618588',
     'SRUN_DEBUG': '3',
     'FTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'ftp_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_NTASKS': '24',
     'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
     'SLURM_STEP_ID': '0',
     'TMUX': '/tmp/tmux-67339/default,43792,2',
     '_CE_CONDA': '',
     'CONDA_PREFIX_1': '/cluster/miniconda3',
     'SLURM_STEP_LAUNCHER_PORT': '33255',
     'SLURM_TASKS_PER_NODE': '24',
     'MAIL': '/var/spool/mail/ye53nis',
     'PATH': '/home/ye53nis/.conda/envs/tf-nightly/bin:/home/lex/Programme/miniconda3/envs/tf-nightly-lab/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
     'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
     'SLURM_JOB_ID': '618588',
     'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tf-nightly',
     'SLURM_JOB_USER': 'ye53nis',
     'SLURM_STEPID': '0',
     'PWD': '/',
     'SLURM_SRUN_COMM_HOST': '192.168.192.5',
     'LANG': 'en_US.UTF-8',
     'SLURM_PTY_WIN_COL': '80',
     'SLURM_UMASK': '0022',
     'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
     'SLURM_JOB_UID': '67339',
     'LOADEDMODULES': '',
     'SLURM_NODEID': '0',
     'TMUX_PANE': '%2',
     'SLURM_SUBMIT_DIR': '/',
     'SLURM_TASK_PID': '75179',
     'SLURM_NPROCS': '24',
     'SLURM_CPUS_ON_NODE': '24',
     'SLURM_DISTRIBUTION': 'block',
     'HTTPS_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'https_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_PROCID': '0',
     'HISTCONTROL': 'ignoredups',
     '_CE_M': '',
     'SLURM_JOB_NODELIST': 'node221',
     'SLURM_PTY_PORT': '41588',
     'HOME': '/home/ye53nis',
     'SHLVL': '3',
     'SLURM_LOCALID': '0',
     'SLURM_JOB_GID': '13280',
     'SLURM_JOB_CPUS_PER_NODE': '24',
     'SLURM_CLUSTER_NAME': 'hpc',
     'no_proxy': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24',
     'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
     'SLURM_SUBMIT_HOST': 'login01',
     'HTTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_JOB_PARTITION': 's_standard',
     'MATHEMATICA_HOME': '/cluster/apps/mathematica/11.3',
     'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
     'LOGNAME': 'ye53nis',
     'SLURM_STEP_NUM_TASKS': '24',
     'QTLIB': '/usr/lib64/qt-3.3/lib',
     'SLURM_JOB_ACCOUNT': 'iaob',
     'SLURM_JOB_NUM_NODES': '1',
     'MODULESHOME': '/usr/share/Modules',
     'CONDA_DEFAULT_ENV': 'tf-nightly',
     'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
     'SLURM_STEP_TASKS_PER_NODE': '24',
     'PORT': '9999','SLURM_STEP_NODELIST': 'node221',
    'DISPLAY': ':0',
    'XDG_RUNTIME_DIR': '',
    'XAUTHORITY': '/home/lex/.Xauthority',
    'BASH_FUNC_module()': '() {  eval `/usr/bin/modulecmd bash $*`\n}',
    '_': '/home/ye53nis/.conda/envs/tf-nightly/bin/jupyter',
    'JPY_PARENT_PID': '94022',
    'CLICOLOR': '1',
    'PAGER': 'cat',
    'GIT_PAGER': 'cat',
    'MPLBACKEND': 'module://ipykernel.pylab.backend_inline'}
    
2.3.2.2 Set mlflow variables
  • mlflow environment variables
    conda activate tf-nightly
    cd /beegfs/ye53nis/drmed-git
    export MLFLOW_EXPERIMENT_NAME=exp-210204-unet
    export MLFLOW_TRACKING_URI=file:./data/mlruns
    mkdir data/exp-210204-unet
    
    (tf-nightly) [ye53nis@node221 drmed-git]$
    
2.3.2.3 run mlflow
  • Use whole dataset (6400 training, 1600 validation, 2000 test), but during training, use only 1/5th of it per epoch
    mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=50 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Nov2020 -P steps_per_epoch=1280 -P validation_steps=320
    
    (tf-nightly) [ye53nis@node221 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=50 -P learning_rate=None -P csv_path=/beegfs/ye53nis/saves/firstartifact_Nov2020 -P steps_per_epoch=1280 -P validation_
    steps=320
    
    INFO: 'exp-210104-unet' does not exist. Creating a new experiment
    
    2021/02/05 18:23:53 INFO mlflow.projects.utils: === Created directory /tmp/tmptojz254v for downloading remote URIs passed to arguments of type 'path' ===
    
    2021/02/05 18:23:53 INFO mlflow.projects.backend.local:
    === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-fd9a200e5a24d4c79a0ff13be73ccb5141ed072c 1>&2
    && python src/fluotracify/training/train.py /beegfs/ye53nis/drmed-git/src 5 0.2 16384 None 50 /beegfs/ye53nis/saves/firstartifact_Nov2020 3 1280 320'
    in run with ID 'b9935d1e554c423fb2852242f4c4504c' ===
    
    2021-02-05 18:24:11.505650: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2021-02-05 18:24:11.505734: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. 2.5.0-dev20210130
    2021-02-05 18:24:48.480544: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2021-02-05 18:24:48.480629: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
    2021-02-05 18:24:48.480678: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node221): /proc/driver/nvidia/version does not exist
    GPUs:  []
    
    train 0 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set007.csv
    train 1 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
    train 2 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set003.csv
    train 3 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set003.csv
    train 4 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set006.csv
    train 5 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set009.csv
    train 6 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set007.csv
    train 7 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set010.csv
    train 8 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set007.csv
    train 9 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set006.csv
    train 10 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set007.csv
    train 11 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set007.csv
    train 12 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set002.csv
    train 13 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set004.csv
    train 14 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set008.csv
    train 15 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set001.csv
    train 16 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set003.csv
    train 17 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set005.csv
    train 18 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set004.csv
    train 19 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set009.csv
    train 20 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set004.csv
    train 21 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set009.csv
    train 22 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set004.csv
    train 23 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set005.csv
    train 24 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set003.csv
    train 25 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set006.csv
    train 26 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set007.csv
    train 27 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set007.csv
    train 28 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set005.csv
    train 29 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set010.csv
    train 30 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set004.csv
    train 31 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set008.csv
    train 32 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set001.csv
    train 33 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set002.csv
    train 34 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set004.csv
    train 35 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set008.csv
    train 36 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set009.csv
    train 37 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set010.csv
    train 38 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set010.csv
    train 39 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
    train 40 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set004.csv
    train 41 /beegfs/ye53nis/saves/firstartifact_Nov2020/subsample/traces_brightclust_Nov2020_D3.0_set004.csv
    train 42 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set002.csv
    train 43 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set007.csv
    train 44 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set003.csv
    train 45 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set002.csv
    train 46 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set005.csv
    train 47 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set006.csv
    train 48 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set008.csv
    train 49 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set001.csv
    train 50 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set009.csv
    train 51 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set006.csv
    train 52 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set006.csv
    train 53 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set005.csv
    train 54 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set001.csv
    train 55 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set005.csv
    train 56 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set009.csv
    train 57 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set006.csv
    train 58 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set001.csv
    train 59 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set002.csv
    train 60 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set006.csv
    train 61 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set004.csv
    train 62 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set003.csv
    train 63 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set001.csv
    train 64 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set002.csv
    train 65 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set009.csv
    train 66 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set008.csv
    train 67 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set001.csv
    train 68 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set002.csv
    train 69 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set003.csv
    train 70 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set005.csv
    train 71 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set005.csv
    train 72 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set004.csv
    train 73 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set010.csv
    train 74 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set010.csv
    train 75 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set010.csv
    train 76 /beegfs/ye53nis/saves/firstartifact_Nov2020/subsample/traces_brightclust_Nov2020_D0.2_set002.csv
    train 77 /beegfs/ye53nis/saves/firstartifact_Nov2020/subsample/traces_brightclust_Nov2020_D1.0_set003.csv
    train 78 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set010.csv
    train 79 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set001.csv
    train 80 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set009.csv
    train 81 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set003.csv
    train 82 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set008.csv
    test 83 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set006.csv
    test 84 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set008.csv
    test 85 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set002.csv
    test 86 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set007.csv
    test 87 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set010.csv
    test 88 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set006.csv
    test 89 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.6/traces_brightclust_Nov2020_D0.6_set009.csv
    test 90 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.08/traces_brightclust_Nov2020_D0.08_set003.csv
    test 91 /beegfs/ye53nis/saves/firstartifact_Nov2020/50/traces_brightclust_Nov2020_D50_set003.csv
    test 92 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set009.csv
    test 93 /beegfs/ye53nis/saves/firstartifact_Nov2020/10/traces_brightclust_Nov2020_D10_set001.csv
    test 94 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set008.csv
    test 95 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.2/traces_brightclust_Nov2020_D0.2_set007.csv
    test 96 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.1/traces_brightclust_Nov2020_D0.1_set002.csv
    test 97 /beegfs/ye53nis/saves/firstartifact_Nov2020/3.0/traces_brightclust_Nov2020_D3.0_set004.csv
    test 98 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.069/traces_brightclust_Nov2020_D0.069_set010.csv
    test 99 /beegfs/ye53nis/saves/firstartifact_Nov2020/subsample/traces_brightclust_Nov2020_D0.069_set001.csv
    test 100 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
    test 101 /beegfs/ye53nis/saves/firstartifact_Nov2020/1.0/traces_brightclust_Nov2020_D1.0_set005.csv
    test 102 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set008.csv
    test 103 /beegfs/ye53nis/saves/firstartifact_Nov2020/0.4/traces_brightclust_Nov2020_D0.4_set005.csv
    
    The given DataFrame was split into 3 parts with shapes: [(16384, 8300), (16384, 8300), (16384, 8300)]
    The given DataFrame was split into 3 parts with shapes: [(16384, 2100), (16384, 2100), (16384, 2100)]
    
    for each 16384 timestap trace there are the following numbers of corrupted timesteps:
    label001_1    5916
    label002_1    7367
    label003_1     954
    label004_1    2965
    label005_1       0
    dtype: int64
    
    2021-02-05 18:31:32.060731: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    
    number of training examples: 6640, number of validation examples: 1660
    
    ------------------------
    number of test examples: 2100
    
    input - shape:   (None, 16384, 1)
    output - shape:  (None, 16384, 1)
    
    2021-02-05 18:31:38.503252: I tensorflow/core/profiler/lib/profiler_session.cc:126] Profiler session initializing.
    2021-02-05 18:31:38.503297: I tensorflow/core/profiler/lib/profiler_session.cc:141] Profiler session started.
    2021-02-05 18:31:38.503361: I tensorflow/core/profiler/lib/profiler_session.cc:158] Profiler session tear down.
    2021/02/05 18:31:38 INFO mlflow.utils.autologging_utils: tensorflow autologging will track hyperparameters, performance metrics, model artifacts, and lineage information for the current tensorflow workflow to the MLflow run with ID 'b9935d1e554c423fb2852242f4c4504c'
    2021/02/05 18:31:38 WARNING mlflow.utils.autologging_utils: MLflow issued a warning during tensorflow autologging: "/home/ye53nis/.conda/envs/mlflow-fd9a200e5a24d4c79a0ff13be73ccb5141ed072c/lib/python3.8/site-packages/mlflow/utils/autologging_utils.py:86:
    UserWarning: Logging to MLflow failed: Changing param values is not allowed. Param with key='batch_size' was already logged with value='5' for run ID='b9935d1e554c423fb2852242f4c4504c'. Attempted logging new value 'None'."
    
    Epoch 1/50
    
    2021-02-05 18:31:49.433599: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:145] None of the MLIR Optimization Passes are enabled (registered 2)
    2021-02-05 18:31:49.686354: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2300000000 Hz
    2021-02-05 18:31:57.757621: I tensorflow/core/profiler/lib/profiler_session.cc:126] Profiler session initializing.
    2021-02-05 18:31:57.757684: I tensorflow/core/profiler/lib/profiler_session.cc:141] Profiler session started.
    2021-02-05 18:31:59.478997: I tensorflow/core/profiler/lib/profiler_session.cc:66] Profiler session collecting data.
    2021-02-05 18:31:59.513272: I tensorflow/core/profiler/lib/profiler_session.cc:158] Profiler session tear down.
    2021-02-05 18:31:59.547737: I tensorflow/core/profiler/rpc/client/save_profile.cc:137] Creating directory: /tmp/tb/train/plugins/profile/2021_02_05_18_31_59
    2021-02-05 18:31:59.563056: I tensorflow/core/profiler/rpc/client/save_profile.cc:143] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2021_02_05_18_31_59/node221.trace.json.gz
    2021-02-05 18:31:59.600514: I tensorflow/core/profiler/rpc/client/save_profile.cc:137] Creating directory: /tmp/tb/train/plugins/profile/2021_02_05_18_31_59
    2021-02-05 18:31:59.600709: I tensorflow/core/profiler/rpc/client/save_profile.cc:143] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2021_02_05_18_31_59/node221.memory_profile.json.gz
    2021-02-05 18:31:59.604669: I tensorflow/core/profiler/rpc/client/capture_profile.cc:251] Creating directory: /tmp/tb/train/plugins/profile/2021_02_05_18_31_59Dumped tool data for xplane.pb to /tmp/tb/train/plugins/profile/2021_02_05_18_31_59/node221.xplane.pb
    Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2021_02_05_18_31_59/node221.overview_page.pb
    Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2021_02_05_18_31_59/node221.input_pipeline.pb
    Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2021_02_05_18_31_59/node221.tensorflow_stats.pb
    Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2021_02_05_18_31_59/node221.kernel_stats.pb
    
    1280/1280 [==============================] - 2294s 2s/step - loss: 0.9535 - tp0.1: 7533469.6347 - fp0.1: 15687196.0695 - tn0.1: 28384947.3810 - fn0.1: 905043.5777 - precision0.1: 0.3192 - recall0.1: 0.8847 - tp0.3: 5299550.3950 - fp0.3
    : 6318015.6128 - tn0.3: 37754137.8610 - fn0.3: 3138962.8173 - precision0.3: 0.4501 - recall0.3: 0.6250 - tp0.5: 3326666.3817 - fp0.5: 1645942.8673 - tn0.5: 42426193.8962 - fn0.5: 5111846.8306 - precision0.5: 0.6413 - recall0.5: 0.3856
    - tp0.7: 2414052.7869 - fp0.7: 649878.6245 - tn0.7: 43422251.9945 - fn0.7: 6024460.4254 - precision0.7: 0.7630 - recall0.7: 0.2684 - tp0.9: 1333571.2881 - fp0.9: 122667.3622 - tn0.9: 43949483.5308 - fn0.9: 7104941.9243 - precision0.9:
    0.8933 - recall0.9: 0.1421 - accuracy: 0.8665 - auc: 0.8370 - val_loss: 4.5065 - val_tp0.1: 3953260.0000 - val_fp0.1: 22248736.0000 - val_tn0.1: 11977.0000 - val_fn0.1: 433.0000 - val_precision0.1: 0.1509 - val_recall0.1: 0.9999 - val_
    tp0.3: 3952790.0000 - val_fp0.3: 22240896.0000 - val_tn0.3: 19810.0000 - val_fn0.3: 903.0000 - val_precision0.3: 0.1509 - val_recall0.3: 0.9998 - val_tp0.5: 3932309.0000 - val_fp0.5: 18000460.0000 - val_tn0.5: 4260247.0000 - val_fn0.5:
     21384.0000 - val_precision0.5: 0.1793 - val_recall0.5: 0.9946 - val_tp0.7: 3907763.0000 - val_fp0.7: 15479875.0000 - val_tn0.7: 6780832.0000 - val_fn0.7: 45930.0000 - val_precision0.7: 0.2016 - val_recall0.7: 0.9884 - val_tp0.9: 38181
    41.0000 - val_fp0.9: 10776068.0000 - val_tn0.9: 11484639.0000 - val_fn0.9: 135552.0000 - val_precision0.9: 0.2616 - val_recall0.9: 0.9657 - val_accuracy: 0.3125 - val_auc: 0.8388
    Epoch 2/50
    1280/1280 [==============================] - 2266s 2s/step - loss: 0.7086 - tp0.1: 7408524.6518 - fp0.1: 9777539.1296 - tn0.1: 34354079.2826 - fn0.1: 970517.5371 - precision0.1: 0.4242 - recall0.1: 0.8837 - tp0.3: 6604603.1468 - fp0.3:
     5177327.2397 - tn0.3: 38954289.9594 - fn0.3: 1774439.0422 - precision0.3: 0.5497 - recall0.3: 0.7873 - tp0.5: 4795568.8735 - fp0.5: 1588506.0874 - tn0.5: 42543101.8407 - fn0.5: 3583473.3154 - precision0.5: 0.7554 - recall0.5: 0.5523 -
     tp0.7: 3842322.5410 - fp0.7: 585624.8493 - tn0.7: 43545999.7440 - fn0.7: 4536719.6479 - precision0.7: 0.8722 - recall0.7: 0.4455 - tp0.9: 2664836.5355 - fp0.9: 183975.7455 - tn0.9: 43947648.4012 - fn0.9: 5714205.6534 - precision0.9: 0
    .9372 - recall0.9: 0.3066 - accuracy: 0.9005 - auc: 0.8983 - val_loss: 8.8910 - val_tp0.1: 4309704.0000 - val_fp0.1: 21433558.0000 - val_tn0.1: 466616.0000 - val_fn0.1: 4509.0000 - val_precision0.1: 0.1674 - val_recall0.1: 0.9990 - val
    _tp0.3: 4304770.0000 - val_fp0.3: 20572086.0000 - val_tn0.3: 1328101.0000 - val_fn0.3: 9443.0000 - val_precision0.3: 0.1730 - val_recall0.3: 0.9978 - val_tp0.5: 4291421.0000 - val_fp0.5: 18429908.0000 - val_tn0.5: 3470280.0000 - val_fn
    0.5: 22792.0000 - val_precision0.5: 0.1889 - val_recall0.5: 0.9947 - val_tp0.7: 4286245.0000 - val_fp0.7: 17960968.0000 - val_tn0.7: 3939218.0000 - val_fn0.7: 27968.0000 - val_precision0.7: 0.1927 - val_recall0.7: 0.9935 - val_tp0.9: 4
    275720.0000 - val_fp0.9: 17168184.0000 - val_tn0.9: 4732003.0000 - val_fn0.9: 38493.0000 - val_precision0.9: 0.1994 - val_recall0.9: 0.9911 - val_accuracy: 0.2961 - val_auc: 0.6353
    Epoch 3/50
    1280/1280 [==============================] - 2269s 2s/step - loss: 0.4737 - tp0.1: 8043601.5324 - fp0.1: 6567924.1288 - tn0.1: 37316662.9188 - fn0.1: 582465.5137 - precision0.1: 0.5472 - recall0.1: 0.9292 - tp0.3: 7343577.1382 - fp0.3:
     2667019.8610 - tn0.3: 41217565.9859 - fn0.3: 1282489.9079 - precision0.3: 0.7338 - recall0.3: 0.8450 - tp0.5: 6431038.6987 - fp0.5: 1096394.8681 - tn0.5: 42788211.8212 - fn0.5: 2195028.3474 - precision0.5: 0.8653 - recall0.5: 0.7272 -
     tp0.7: 5761648.7400 - fp0.7: 554569.4169 - tn0.7: 43330028.4895 - fn0.7: 2864418.3060 - precision0.7: 0.9155 - recall0.7: 0.6553 - tp0.9: 4604418.9586 - fp0.9: 201930.2724 - tn0.9: 43682665.5402 - fn0.9: 4021648.0874 - precision0.9: 0
    .9590 - recall0.9: 0.5232 - accuracy: 0.9363 - auc: 0.9445 - val_loss: 0.6077 - val_tp0.1: 3651028.0000 - val_fp0.1: 3375728.0000 - val_tn0.1: 18695272.0000 - val_fn0.1: 492376.0000 - val_precision0.1: 0.5196 - val_recall0.1: 0.8812 -
    val_tp0.3: 3024227.0000 - val_fp0.3: 1523943.0000 - val_tn0.3: 20547044.0000 - val_fn0.3: 1119177.0000 - val_precision0.3: 0.6649 - val_recall0.3: 0.7299 - val_tp0.5: 2511712.0000 - val_fp0.5: 734684.0000 - val_tn0.5: 21336306.0000 - v
    al_fn0.5: 1631692.0000 - val_precision0.5: 0.7737 - val_recall0.5: 0.6062 - val_tp0.7: 2014285.0000 - val_fp0.7: 342510.0000 - val_tn0.7: 21728484.0000 - val_fn0.7: 2129119.0000 - val_precision0.7: 0.8547 - val_recall0.7: 0.4861 - val_
    tp0.9: 1375650.0000 - val_fp0.9: 83494.0000 - val_tn0.9: 21987508.0000 - val_fn0.9: 2767754.0000 - val_precision0.9: 0.9428 - val_recall0.9: 0.3320 - val_accuracy: 0.9097 - val_auc: 0.9057
    Epoch 4/50
    1280/1280 [==============================] - 2269s 2s/step - loss: 0.4104 - tp0.1: 8058950.6292 - fp0.1: 5549150.7541 - tn0.1: 38382155.2373 - fn0.1: 520398.2319 - precision0.1: 0.5890 - recall0.1: 0.9388 - tp0.3: 7465362.4442 - fp0.3:
     2368717.0968 - tn0.3: 41562596.7416 - fn0.3: 1113986.4169 - precision0.3: 0.7555 - recall0.3: 0.8690 - tp0.5: 6948225.4496 - fp0.5: 1288524.5277 - tn0.5: 42642762.4083 - fn0.5: 1631123.4114 - precision0.5: 0.8408 - recall0.5: 0.8075 -
     tp0.7: 6012711.4176 - fp0.7: 481306.9664 - tn0.7: 43449995.4871 - fn0.7: 2566637.4434 - precision0.7: 0.9250 - recall0.7: 0.6949 - tp0.9: 4953486.9508 - fp0.9: 170919.6136 - tn0.9: 43760406.4403 - fn0.9: 3625861.9102 - precision0.9: 0
    .9659 - recall0.9: 0.5674 - accuracy: 0.9430 - auc: 0.9540 - val_loss: 0.7360 - val_tp0.1: 3518565.0000 - val_fp0.1: 2418501.0000 - val_tn0.1: 19514842.0000 - val_fn0.1: 762498.0000 - val_precision0.1: 0.5926 - val_recall0.1: 0.8219 -
    val_tp0.3: 2934643.0000 - val_fp0.3: 1727197.0000 - val_tn0.3: 20206140.0000 - val_fn0.3: 1346420.0000 - val_precision0.3: 0.6295 - val_recall0.3: 0.6855 - val_tp0.5: 2401516.0000 - val_fp0.5: 1120115.0000 - val_tn0.5: 20813218.0000 -
    val_fn0.5: 1879547.0000 - val_precision0.5: 0.6819 - val_recall0.5: 0.5610 - val_tp0.7: 1995881.0000 - val_fp0.7: 587480.0000 - val_tn0.7: 21345854.0000 - val_fn0.7: 2285182.0000 - val_precision0.7: 0.7726 - val_recall0.7: 0.4662 - val
    _tp0.9: 1458232.0000 - val_fp0.9: 213624.0000 - val_tn0.9: 21719718.0000 - val_fn0.9: 2822831.0000 - val_precision0.9: 0.8722 - val_recall0.9: 0.3406 - val_accuracy: 0.8856 - val_auc: 0.8746
    Epoch 5/50
    1280/1280 [==============================] - 2256s 2s/step - loss: 0.3330 - tp0.1: 7934527.3903 - fp0.1: 4492667.3357 - tn0.1: 39650554.1023 - fn0.1: 432898.7760 - precision0.1: 0.6303 - recall0.1: 0.9481 - tp0.3: 7499415.2927 - fp0.3:
     1967988.6995 - tn0.3: 42175278.8080 - fn0.3: 868010.8735 - precision0.3: 0.7854 - recall0.3: 0.8951 - tp0.5: 7067133.4980 - fp0.5: 1049951.0960 - tn0.5: 43093262.6339 - fn0.5: 1300292.6682 - precision0.5: 0.8642 - recall0.5: 0.8435 -
    tp0.7: 6367827.1046 - fp0.7: 438987.0765 - tn0.7: 43704237.3872 - fn0.7: 1999599.0617 - precision0.7: 0.9322 - recall0.7: 0.7564 - tp0.9: 5357606.0351 - fp0.9: 138722.6315 - tn0.9: 44004504.8205 - fn0.9: 3009820.1311 - precision0.9: 0.
    9715 - recall0.9: 0.6358 - accuracy: 0.9541 - auc: 0.9638 - val_loss: 0.7327 - val_tp0.1: 3649063.0000 - val_fp0.1: 3036718.0000 - val_tn0.1: 18938916.0000 - val_fn0.1: 589701.0000 - val_precision0.1: 0.5458 - val_recall0.1: 0.8609 - v
    al_tp0.3: 3283999.0000 - val_fp0.3: 2440985.0000 - val_tn0.3: 19534644.0000 - val_fn0.3: 954765.0000 - val_precision0.3: 0.5736 - val_recall0.3: 0.7748 - val_tp0.5: 2951063.0000 - val_fp0.5: 2094966.0000 - val_tn0.5: 19880668.0000 - va
    l_fn0.5: 1287701.0000 - val_precision0.5: 0.5848 - val_recall0.5: 0.6962 - val_tp0.7: 2702547.0000 - val_fp0.7: 1708012.0000 - val_tn0.7: 20267626.0000 - val_fn0.7: 1536217.0000 - val_precision0.7: 0.6127 - val_recall0.7: 0.6376 - val_
    tp0.9: 2269870.0000 - val_fp0.9: 882080.0000 - val_tn0.9: 21093558.0000 - val_fn0.9: 1968894.0000 - val_precision0.9: 0.7201 - val_recall0.9: 0.5355 - val_accuracy: 0.8710 - val_auc: 0.8874
    Epoch 6/50
    1280/1280 [==============================] - 2267s 2s/step - loss: 0.3148 - tp0.1: 8107422.3794 - fp0.1: 4149459.1678 - tn0.1: 39840764.2201 - fn0.1: 412995.5535 - precision0.1: 0.6568 - recall0.1: 0.9505 - tp0.3: 7702537.1499 - fp0.3:
     1842434.6089 - tn0.3: 42147804.7213 - fn0.3: 817880.7830 - precision0.3: 0.8055 - recall0.3: 0.9021 - tp0.5: 7257172.3966 - fp0.5: 947340.6534 - tn0.5: 43042899.9469 - fn0.5: 1263245.5363 - precision0.5: 0.8838 - recall0.5: 0.8495 - t
    p0.7: 6767789.5652 - fp0.7: 485240.0351 - tn0.7: 43504992.3216 - fn0.7: 1752628.3677 - precision0.7: 0.9324 - recall0.7: 0.7924 - tp0.9: 5789730.9688 - fp0.9: 133595.3934 - tn0.9: 43856655.0180 - fn0.9: 2730686.9641 - precision0.9: 0.9
    771 - recall0.9: 0.6757 - accuracy: 0.9572 - auc: 0.9671 - val_loss: 0.5438 - val_tp0.1: 3469770.0000 - val_fp0.1: 1783021.0000 - val_tn0.1: 20362438.0000 - val_fn0.1: 599169.0000 - val_precision0.1: 0.6606 - val_recall0.1: 0.8527 - va
    l_tp0.3: 2981687.0000 - val_fp0.3: 776413.0000 - val_tn0.3: 21369044.0000 - val_fn0.3: 1087252.0000 - val_precision0.3: 0.7934 - val_recall0.3: 0.7328 - val_tp0.5: 2491815.0000 - val_fp0.5: 326150.0000 - val_tn0.5: 21819316.0000 - val_
    fn0.5: 1577124.0000 - val_precision0.5: 0.8843 - val_recall0.5: 0.6124 - val_tp0.7: 2092865.0000 - val_fp0.7: 128692.0000 - val_tn0.7: 22016760.0000 - val_fn0.7: 1976074.0000 - val_precision0.7: 0.9421 - val_recall0.7: 0.5144 - val_tp0
    .9: 1584013.0000 - val_fp0.9: 18793.0000 - val_tn0.9: 22126662.0000 - val_fn0.9: 2484926.0000 - val_precision0.9: 0.9883 - val_recall0.9: 0.3893 - val_accuracy: 0.9274 - val_auc: 0.9112
    Epoch 7/50
    1280/1280 [==============================] - 2250s 2s/step - loss: 0.2752 - tp0.1: 8215921.6979 - fp0.1: 3662051.7354 - tn0.1: 40263427.6151 - fn0.1: 369255.7354 - precision0.1: 0.6930 - recall0.1: 0.9578 - tp0.3: 7873789.5113 - fp0.3:
     1775915.5043 - tn0.3: 42149585.2553 - fn0.3: 711387.9219 - precision0.3: 0.8180 - recall0.3: 0.9178 - tp0.5: 7421898.0468 - fp0.5: 864234.9992 - tn0.5: 43061238.2006 - fn0.5: 1163279.3864 - precision0.5: 0.8970 - recall0.5: 0.8653 - t
    p0.7: 6899341.9555 - fp0.7: 403014.4684 - tn0.7: 43522441.7424 - fn0.7: 1685835.4778 - precision0.7: 0.9455 - recall0.7: 0.8048 - tp0.9: 6047520.0070 - fp0.9: 123672.6019 - tn0.9: 43801830.9766 - fn0.9: 2537657.4262 - precision0.9: 0.9
    803 - recall0.9: 0.7036 - accuracy: 0.9616 - auc: 0.9725 - val_loss: 0.5870 - val_tp0.1: 3992712.0000 - val_fp0.1: 3100961.0000 - val_tn0.1: 18785236.0000 - val_fn0.1: 335494.0000 - val_precision0.1: 0.5629 - val_recall0.1: 0.9225 - va
    l_tp0.3: 3697613.0000 - val_fp0.3: 2362999.0000 - val_tn0.3: 19523192.0000 - val_fn0.3: 630593.0000 - val_precision0.3: 0.6101 - val_recall0.3: 0.8543 - val_tp0.5: 3356073.0000 - val_fp0.5: 1926159.0000 - val_tn0.5: 19960034.0000 - val
    _fn0.5: 972133.0000 - val_precision0.5: 0.6354 - val_recall0.5: 0.7754 - val_tp0.7: 3025256.0000 - val_fp0.7: 1584253.0000 - val_tn0.7: 20301946.0000 - val_fn0.7: 1302950.0000 - val_precision0.7: 0.6563 - val_recall0.7: 0.6990 - val_tp
    0.9: 2583002.0000 - val_fp0.9: 1016388.0000 - val_tn0.9: 20869810.0000 - val_fn0.9: 1745204.0000 - val_precision0.9: 0.7176 - val_recall0.9: 0.5968 - val_accuracy: 0.8894 - val_auc: 0.9219
    Epoch 8/50
    1280/1280 [==============================] - 2264s 2s/step - loss: 0.2659 - tp0.1: 8145147.1616 - fp0.1: 3529957.8525 - tn0.1: 40486117.0164 - fn0.1: 349436.3614 - precision0.1: 0.6977 - recall0.1: 0.9588 - tp0.3: 7816330.9524 - fp0.3:
     1688486.8493 - tn0.3: 42327603.8790 - fn0.3: 678252.5706 - precision0.3: 0.8221 - recall0.3: 0.9201 - tp0.5: 7396513.9524 - fp0.5: 842774.1983 - tn0.5: 43173300.2077 - fn0.5: 1098069.5706 - precision0.5: 0.8972 - recall0.5: 0.8704 - t
    p0.7: 6908332.2162 - fp0.7: 399714.5441 - tn0.7: 43616338.9617 - fn0.7: 1586251.3068 - precision0.7: 0.9448 - recall0.7: 0.8128 - tp0.9: 6106920.7744 - fp0.9: 123699.0000 - tn0.9: 43892389.6027 - fn0.9: 2387662.7486 - precision0.9: 0.9
    796 - recall0.9: 0.7174 - accuracy: 0.9627 - auc: 0.9735 - val_loss: 0.5059 - val_tp0.1: 3832991.0000 - val_fp0.1: 2975127.0000 - val_tn0.1: 19105348.0000 - val_fn0.1: 300933.0000 - val_precision0.1: 0.5630 - val_recall0.1: 0.9272 - va
    l_tp0.3: 3565080.0000 - val_fp0.3: 2148056.0000 - val_tn0.3: 19932426.0000 - val_fn0.3: 568844.0000 - val_precision0.3: 0.6240 - val_recall0.3: 0.8624 - val_tp0.5: 3244053.0000 - val_fp0.5: 1637048.0000 - val_tn0.5: 20443428.0000 - val
    _fn0.5: 889871.0000 - val_precision0.5: 0.6646 - val_recall0.5: 0.7847 - val_tp0.7: 2923895.0000 - val_fp0.7: 1240875.0000 - val_tn0.7: 20839598.0000 - val_fn0.7: 1210029.0000 - val_precision0.7: 0.7021 - val_recall0.7: 0.7073 - val_tp
    0.9: 2501710.0000 - val_fp0.9: 630185.0000 - val_tn0.9: 21450288.0000 - val_fn0.9: 1632214.0000 - val_precision0.9: 0.7988 - val_recall0.9: 0.6052 - val_accuracy: 0.9036 - val_auc: 0.9333
    Epoch 9/50
    1280/1280 [==============================] - 2265s 2s/step - loss: 0.2721 - tp0.1: 8177633.5020 - fp0.1: 3535203.1725 - tn0.1: 40418482.1335 - fn0.1: 379327.7697 - precision0.1: 0.7002 - recall0.1: 0.9554 - tp0.3: 7847088.7190 - fp0.3:
     1695397.4301 - tn0.3: 42258294.9282 - fn0.3: 709872.5527 - precision0.3: 0.8241 - recall0.3: 0.9169 - tp0.5: 7427192.4395 - fp0.5: 834997.2678 - tn0.5: 43118702.8587 - fn0.5: 1129768.8322 - precision0.5: 0.9006 - recall0.5: 0.8679 - t
    p0.7: 6938678.2849 - fp0.7: 395857.1007 - tn0.7: 43557853.7744 - fn0.7: 1618282.9867 - precision0.7: 0.9471 - recall0.7: 0.8106 - tp0.9: 6133368.2951 - fp0.9: 120955.7010 - tn0.9: 43832745.2506 - fn0.9: 2423592.9766 - precision0.9: 0.9
    811 - recall0.9: 0.7170 - accuracy: 0.9631 - auc: 0.9718 - val_loss: 0.5832 - val_tp0.1: 3893714.0000 - val_fp0.1: 3303616.0000 - val_tn0.1: 18681092.0000 - val_fn0.1: 335977.0000 - val_precision0.1: 0.5410 - val_recall0.1: 0.9206 - va
    l_tp0.3: 3627551.0000 - val_fp0.3: 2498540.0000 - val_tn0.3: 19486168.0000 - val_fn0.3: 602140.0000 - val_precision0.3: 0.5921 - val_recall0.3: 0.8576 - val_tp0.5: 3322248.0000 - val_fp0.5: 1987125.0000 - val_tn0.5: 19997588.0000 - val
    _fn0.5: 907443.0000 - val_precision0.5: 0.6257 - val_recall0.5: 0.7855 - val_tp0.7: 3013599.0000 - val_fp0.7: 1581768.0000 - val_tn0.7: 20402948.0000 - val_fn0.7: 1216092.0000 - val_precision0.7: 0.6558 - val_recall0.7: 0.7125 - val_tp
    0.9: 2613832.0000 - val_fp0.9: 924572.0000 - val_tn0.9: 21060124.0000 - val_fn0.9: 1615859.0000 - val_precision0.9: 0.7387 - val_recall0.9: 0.6180 - val_accuracy: 0.8896 - val_auc: 0.9233
    Epoch 10/50
    1280/1280 [==============================] - 2167s 2s/step - loss: 0.2587 - tp0.1: 8173202.1491 - fp0.1: 3419344.5308 - tn0.1: 40568773.3271 - fn0.1: 349346.8673 - precision0.1: 0.7050 - recall0.1: 0.9592 - tp0.3: 7845280.8431 - fp0.3:
     1633368.1054 - tn0.3: 42354746.7455 - fn0.3: 677268.1733 - precision0.3: 0.8274 - recall0.3: 0.9210 - tp0.5: 7444574.4614 - fp0.5: 808848.8946 - tn0.5: 43179274.2358 - fn0.5: 1077974.5550 - precision0.5: 0.9017 - recall0.5: 0.8744 - t
    p0.7: 6974399.5706 - fp0.7: 382341.6963 - tn0.7: 43605795.4044 - fn0.7: 1548149.4457 - precision0.7: 0.9478 - recall0.7: 0.8197 - tp0.9: 6212727.7385 - fp0.9: 117973.7447 - tn0.9: 43870128.1835 - fn0.9: 2309821.2779 - precision0.9: 0.9
    813 - recall0.9: 0.7300 - accuracy: 0.9641 - auc: 0.9742 - val_loss: 0.6408 - val_tp0.1: 3921454.0000 - val_fp0.1: 3165544.0000 - val_tn0.1: 18831180.0000 - val_fn0.1: 296224.0000 - val_precision0.1: 0.5533 - val_recall0.1: 0.9298 - va
    l_tp0.3: 3674710.0000 - val_fp0.3: 2541189.0000 - val_tn0.3: 19455532.0000 - val_fn0.3: 542968.0000 - val_precision0.3: 0.5912 - val_recall0.3: 0.8713 - val_tp0.5: 3370841.0000 - val_fp0.5: 2184413.0000 - val_tn0.5: 19812308.0000 - val
    _fn0.5: 846837.0000 - val_precision0.5: 0.6068 - val_recall0.5: 0.7992 - val_tp0.7: 3057156.0000 - val_fp0.7: 1929320.0000 - val_tn0.7: 20067408.0000 - val_fn0.7: 1160522.0000 - val_precision0.7: 0.6131 - val_recall0.7: 0.7248 - val_tp
    0.9: 2691235.0000 - val_fp0.9: 1545100.0000 - val_tn0.9: 20451626.0000 - val_fn0.9: 1526443.0000 - val_precision0.9: 0.6353 - val_recall0.9: 0.6381 - val_accuracy: 0.8844 - val_auc: 0.9185
    Epoch 11/50
    1280/1280 [==============================] - 2195s 2s/step - loss: 0.2597 - tp0.1: 8080787.5496 - fp0.1: 3464173.5402 - tn0.1: 40607994.5691 - fn0.1: 357720.7119 - precision0.1: 0.6992 - recall0.1: 0.9583 - tp0.3: 7754354.7728 - fp0.3:
     1652487.9594 - tn0.3: 42419634.1717 - fn0.3: 684153.4887 - precision0.3: 0.8245 - recall0.3: 0.9197 - tp0.5: 7350582.6308 - fp0.5: 814018.4028 - tn0.5: 43258092.7291 - fn0.5: 1087925.6308 - precision0.5: 0.9008 - recall0.5: 0.8720 - t
    p0.7: 6874278.2834 - fp0.7: 383492.2842 - tn0.7: 43688666.7549 - fn0.7: 1564229.9781 - precision0.7: 0.9477 - recall0.7: 0.8158 - tp0.9: 6117529.6448 - fp0.9: 121059.4980 - tn0.9: 43951099.9110 - fn0.9: 2320978.6167 - precision0.9: 0.9
    810 - recall0.9: 0.7266 - accuracy: 0.9642 - auc: 0.9736 - val_loss: 0.6007 - val_tp0.1: 3926588.0000 - val_fp0.1: 3104163.0000 - val_tn0.1: 18816186.0000 - val_fn0.1: 367465.0000 - val_precision0.1: 0.5585 - val_recall0.1: 0.9144 - va
    l_tp0.3: 3638712.0000 - val_fp0.3: 2431931.0000 - val_tn0.3: 19488412.0000 - val_fn0.3: 655341.0000 - val_precision0.3: 0.5994 - val_recall0.3: 0.8474 - val_tp0.5: 3274020.0000 - val_fp0.5: 1937942.0000 - val_tn0.5: 19982404.0000 - val
    _fn0.5: 1020033.0000 - val_precision0.5: 0.6282 - val_recall0.5: 0.7625 - val_tp0.7: 2918102.0000 - val_fp0.7: 1553208.0000 - val_tn0.7: 20367136.0000 - val_fn0.7: 1375951.0000 - val_precision0.7: 0.6526 - val_recall0.7: 0.6796 - val_t
    p0.9: 2486914.0000 - val_fp0.9: 949847.0000 - val_tn0.9: 20970500.0000 - val_fn0.9: 1807139.0000 - val_precision0.9: 0.7236 - val_recall0.9: 0.5792 - val_accuracy: 0.8872 - val_auc: 0.9170
    Epoch 12/50
    1280/1280 [==============================] - 2210s 2s/step - loss: 0.2529 - tp0.1: 8177778.9118 - fp0.1: 3446344.3888 - tn0.1: 40554141.6956 - fn0.1: 332395.2459 - precision0.1: 0.7024 - recall0.1: 0.9615 - tp0.3: 7866160.7268 - fp0.3:
     1663533.3247 - tn0.3: 42336942.1569 - fn0.3: 644013.4309 - precision0.3: 0.8244 - recall0.3: 0.9251 - tp0.5: 7468651.7650 - fp0.5: 826902.3630 - tn0.5: 43173597.9063 - fn0.5: 1041522.3927 - precision0.5: 0.8996 - recall0.5: 0.8783 - t
    p0.7: 6982137.1124 - fp0.7: 384836.8439 - tn0.7: 43615637.7978 - fn0.7: 1528037.0453 - precision0.7: 0.9473 - recall0.7: 0.8206 - tp0.9: 6215436.7151 - fp0.9: 115782.2108 - tn0.9: 43884692.7596 - fn0.9: 2294737.4426 - precision0.9: 0.9
    816 - recall0.9: 0.7291 - accuracy: 0.9644 - auc: 0.9753 - val_loss: 0.5433 - val_tp0.1: 3996675.0000 - val_fp0.1: 3049375.0000 - val_tn0.1: 18853738.0000 - val_fn0.1: 314612.0000 - val_precision0.1: 0.5672 - val_recall0.1: 0.9270 - va
    l_tp0.3: 3730606.0000 - val_fp0.3: 2336826.0000 - val_tn0.3: 19566286.0000 - val_fn0.3: 580681.0000 - val_precision0.3: 0.6149 - val_recall0.3: 0.8653 - val_tp0.5: 3416555.0000 - val_fp0.5: 1856264.0000 - val_tn0.5: 20046848.0000 - val
    _fn0.5: 894732.0000 - val_precision0.5: 0.6480 - val_recall0.5: 0.7925 - val_tp0.7: 3097348.0000 - val_fp0.7: 1461441.0000 - val_tn0.7: 20441664.0000 - val_fn0.7: 1213939.0000 - val_precision0.7: 0.6794 - val_recall0.7: 0.7184 - val_tp
    0.9: 2698173.0000 - val_fp0.9: 854641.0000 - val_tn0.9: 21048466.0000 - val_fn0.9: 1613114.0000 - val_precision0.9: 0.7594 - val_recall0.9: 0.6258 - val_accuracy: 0.8951 - val_auc: 0.9303
    Epoch 13/50
    1280/1280 [==============================] - 2209s 2s/step - loss: 0.2538 - tp0.1: 8165544.5761 - fp0.1: 3360321.7119 - tn0.1: 40640034.6550 - fn0.1: 344752.3396 - precision0.1: 0.7106 - recall0.1: 0.9599 - tp0.3: 7853592.0312 - fp0.3:
     1646282.5488 - tn0.3: 42354071.1335 - fn0.3: 656704.8845 - precision0.3: 0.8279 - recall0.3: 0.9236 - tp0.5: 7458424.3138 - fp0.5: 826341.5082 - tn0.5: 43174017.1226 - fn0.5: 1051872.6019 - precision0.5: 0.9009 - recall0.5: 0.8775 - t
    p0.7: 6979820.3271 - fp0.7: 395956.8899 - tn0.7: 43604407.7775 - fn0.7: 1530476.5886 - precision0.7: 0.9468 - recall0.7: 0.8215 - tp0.9: 6232427.8197 - fp0.9: 129181.2912 - tn0.9: 43871195.4387 - fn0.9: 2277869.0960 - precision0.9: 0.9
    799 - recall0.9: 0.7346 - accuracy: 0.9644 - auc: 0.9747 - val_loss: 0.5485 - val_tp0.1: 3997124.0000 - val_fp0.1: 3403162.0000 - val_tn0.1: 18545812.0000 - val_fn0.1: 268302.0000 - val_precision0.1: 0.5401 - val_recall0.1: 0.9371 - va
    l_tp0.3: 3749177.0000 - val_fp0.3: 2616473.0000 - val_tn0.3: 19332510.0000 - val_fn0.3: 516249.0000 - val_precision0.3: 0.5890 - val_recall0.3: 0.8790 - val_tp0.5: 3440083.0000 - val_fp0.5: 2075288.0000 - val_tn0.5: 19873688.0000 - val
    _fn0.5: 825343.0000 - val_precision0.5: 0.6237 - val_recall0.5: 0.8065 - val_tp0.7: 3117260.0000 - val_fp0.7: 1623393.0000 - val_tn0.7: 20325572.0000 - val_fn0.7: 1148166.0000 - val_precision0.7: 0.6576 - val_recall0.7: 0.7308 - val_tp
    0.9: 2708258.0000 - val_fp0.9: 926738.0000 - val_tn0.9: 21022226.0000 - val_fn0.9: 1557168.0000 - val_precision0.9: 0.7451 - val_recall0.9: 0.6349 - val_accuracy: 0.8894 - val_auc: 0.9322
    Epoch 14/50
    1280/1280 [==============================] - 2223s 2s/step - loss: 0.2483 - tp0.1: 8105005.2256 - fp0.1: 3301412.7728 - tn0.1: 40770974.7994 - fn0.1: 333249.2350 - precision0.1: 0.7102 - recall0.1: 0.9606 - tp0.3: 7797681.4614 - fp0.3:
     1632480.2381 - tn0.3: 42439931.4738 - fn0.3: 640572.9992 - precision0.3: 0.8273 - recall0.3: 0.9241 - tp0.5: 7399965.9297 - fp0.5: 804918.4660 - tn0.5: 43267490.8134 - fn0.5: 1038288.5308 - precision0.5: 0.9020 - recall0.5: 0.8772 - t
    p0.7: 6921871.3716 - fp0.7: 370916.4372 - tn0.7: 43701494.4621 - fn0.7: 1516383.0890 - precision0.7: 0.9493 - recall0.7: 0.8208 - tp0.9: 6187238.5597 - fp0.9: 114174.0468 - tn0.9: 43958233.5792 - fn0.9: 2251015.9009 - precision0.9: 0.9
    819 - recall0.9: 0.7337 - accuracy: 0.9651 - auc: 0.9752 - val_loss: 0.4990 - val_tp0.1: 3920222.0000 - val_fp0.1: 2895972.0000 - val_tn0.1: 19106396.0000 - val_fn0.1: 291803.0000 - val_precision0.1: 0.5751 - val_recall0.1: 0.9307 - va
    l_tp0.3: 3647799.0000 - val_fp0.3: 2161341.0000 - val_tn0.3: 19841040.0000 - val_fn0.3: 564226.0000 - val_precision0.3: 0.6279 - val_recall0.3: 0.8660 - val_tp0.5: 3322418.0000 - val_fp0.5: 1638347.0000 - val_tn0.5: 20364020.0000 - val
    _fn0.5: 889607.0000 - val_precision0.5: 0.6697 - val_recall0.5: 0.7888 - val_tp0.7: 2985353.0000 - val_fp0.7: 1222889.0000 - val_tn0.7: 20779488.0000 - val_fn0.7: 1226672.0000 - val_precision0.7: 0.7094 - val_recall0.7: 0.7088 - val_tp
    0.9: 2563095.0000 - val_fp0.9: 657853.0000 - val_tn0.9: 21344516.0000 - val_fn0.9: 1648930.0000 - val_precision0.9: 0.7958 - val_recall0.9: 0.6085 - val_accuracy: 0.9036 - val_auc: 0.9356
    Epoch 15/50
    1280/1280 [==============================] - 2223s 2s/step - loss: 0.2505 - tp0.1: 8129718.4130 - fp0.1: 3285117.6674 - tn0.1: 40755981.7447 - fn0.1: 339847.1585 - precision0.1: 0.7126 - recall0.1: 0.9599 - tp0.3: 7829732.9274 - fp0.3:
     1632416.1491 - tn0.3: 42408661.6440 - fn0.3: 639832.6440 - precision0.3: 0.8279 - recall0.3: 0.9244 - tp0.5: 7437810.6112 - fp0.5: 808981.5230 - tn0.5: 43232126.9953 - fn0.5: 1031754.9602 - precision0.5: 0.9024 - recall0.5: 0.8777 - t
    p0.7: 6957910.6557 - fp0.7: 373871.2498 - tn0.7: 43667250.3630 - fn0.7: 1511654.9157 - precision0.7: 0.9493 - recall0.7: 0.8208 - tp0.9: 6212919.2295 - fp0.9: 115091.8977 - tn0.9: 43925968.1874 - fn0.9: 2256646.3419 - precision0.9: 0.9
    817 - recall0.9: 0.7317 - accuracy: 0.9649 - auc: 0.9750 - val_loss: 0.5413 - val_tp0.1: 3929730.0000 - val_fp0.1: 3234417.0000 - val_tn0.1: 18748682.0000 - val_fn0.1: 301573.0000 - val_precision0.1: 0.5485 - val_recall0.1: 0.9287 - va
    l_tp0.3: 3652364.0000 - val_fp0.3: 2547235.0000 - val_tn0.3: 19435862.0000 - val_fn0.3: 578939.0000 - val_precision0.3: 0.5891 - val_recall0.3: 0.8632 - val_tp0.5: 3294284.0000 - val_fp0.5: 1952001.0000 - val_tn0.5: 20031108.0000 - val
    _fn0.5: 937019.0000 - val_precision0.5: 0.6279 - val_recall0.5: 0.7786 - val_tp0.7: 2934269.0000 - val_fp0.7: 1461182.0000 - val_tn0.7: 20521920.0000 - val_fn0.7: 1297034.0000 - val_precision0.7: 0.6676 - val_recall0.7: 0.6935 - val_tp
    0.9: 2494237.0000 - val_fp0.9: 748300.0000 - val_tn0.9: 21234804.0000 - val_fn0.9: 1737066.0000 - val_precision0.9: 0.7692 - val_recall0.9: 0.5895 - val_accuracy: 0.8898 - val_auc: 0.9282
    Epoch 16/50
    1280/1280 [==============================] - 2215s 2s/step - loss: 0.2478 - tp0.1: 8105796.2350 - fp0.1: 3240299.5347 - tn0.1: 40826166.4356 - fn0.1: 338386.8126 - precision0.1: 0.7132 - recall0.1: 0.9597 - tp0.3: 7813940.3685 - fp0.3:
     1622845.5285 - tn0.3: 42443650.8525 - fn0.3: 630242.6792 - precision0.3: 0.8274 - recall0.3: 0.9252 - tp0.5: 7422996.8228 - fp0.5: 793661.0414 - tn0.5: 43272824.2966 - fn0.5: 1021186.2248 - precision0.5: 0.9030 - recall0.5: 0.8785 - t
    p0.7: 6948059.6292 - fp0.7: 366674.7080 - tn0.7: 43699824.9578 - fn0.7: 1496123.4184 - precision0.7: 0.9495 - recall0.7: 0.8219 - tp0.9: 6225846.1991 - fp0.9: 114013.1436 - tn0.9: 43952461.9219 - fn0.9: 2218336.8486 - precision0.9: 0.9
    817 - recall0.9: 0.7370 - accuracy: 0.9654 - auc: 0.9750 - val_loss: 0.5514 - val_tp0.1: 3985944.0000 - val_fp0.1: 3204062.0000 - val_tn0.1: 18714132.0000 - val_fn0.1: 310258.0000 - val_precision0.1: 0.5544 - val_recall0.1: 0.9278 - va
    l_tp0.3: 3706293.0000 - val_fp0.3: 2449108.0000 - val_tn0.3: 19469088.0000 - val_fn0.3: 589909.0000 - val_precision0.3: 0.6021 - val_recall0.3: 0.8627 - val_tp0.5: 3369179.0000 - val_fp0.5: 1882360.0000 - val_tn0.5: 20035840.0000 - val
    _fn0.5: 927023.0000 - val_precision0.5: 0.6416 - val_recall0.5: 0.7842 - val_tp0.7: 3033922.0000 - val_fp0.7: 1434846.0000 - val_tn0.7: 20483356.0000 - val_fn0.7: 1262280.0000 - val_precision0.7: 0.6789 - val_recall0.7: 0.7062 - val_tp
    0.9: 2621404.0000 - val_fp0.9: 840233.0000 - val_tn0.9: 21077966.0000 - val_fn0.9: 1674798.0000 - val_precision0.9: 0.7573 - val_recall0.9: 0.6102 - val_accuracy: 0.8928 - val_auc: 0.9287
    Epoch 17/50
    1280/1280 [==============================] - 2214s 2s/step - loss: 0.2484 - tp0.1: 8227665.5144 - fp0.1: 3202591.7237 - tn0.1: 40740266.8517 - fn0.1: 340122.5121 - precision0.1: 0.7214 - recall0.1: 0.9603 - tp0.3: 7941320.1460 - fp0.3:
     1633513.9258 - tn0.3: 42309362.5909 - fn0.3: 626467.8806 - precision0.3: 0.8301 - recall0.3: 0.9272 - tp0.5: 7558350.9258 - fp0.5: 804203.6698 - tn0.5: 43138639.2201 - fn0.5: 1009437.1007 - precision0.5: 0.9041 - recall0.5: 0.8831 - t
    p0.7: 7083529.7642 - fp0.7: 372694.8454 - tn0.7: 43570150.8829 - fn0.7: 1484258.2623 - precision0.7: 0.9501 - recall0.7: 0.8284 - tp0.9: 6341241.1249 - fp0.9: 113794.4754 - tn0.9: 43829043.6105 - fn0.9: 2226546.9016 - precision0.9: 0.9
    824 - recall0.9: 0.7414 - accuracy: 0.9652 - auc: 0.9756 - val_loss: 0.5142 - val_tp0.1: 3628279.0000 - val_fp0.1: 2753615.0000 - val_tn0.1: 19485962.0000 - val_fn0.1: 346542.0000 - val_precision0.1: 0.5685 - val_recall0.1: 0.9128 - va
    l_tp0.3: 3324088.0000 - val_fp0.3: 2056425.0000 - val_tn0.3: 20183156.0000 - val_fn0.3: 650733.0000 - val_precision0.3: 0.6178 - val_recall0.3: 0.8363 - val_tp0.5: 2972065.0000 - val_fp0.5: 1467191.0000 - val_tn0.5: 20772396.0000 - val
    _fn0.5: 1002756.0000 - val_precision0.5: 0.6695 - val_recall0.5: 0.7477 - val_tp0.7: 2650974.0000 - val_fp0.7: 1083574.0000 - val_tn0.7: 21156004.0000 - val_fn0.7: 1323847.0000 - val_precision0.7: 0.7099 - val_recall0.7: 0.6669 - val_t
    p0.9: 2300149.0000 - val_fp0.9: 585684.0000 - val_tn0.9: 21653888.0000 - val_fn0.9: 1674672.0000 - val_precision0.9: 0.7970 - val_recall0.9: 0.5787 - val_accuracy: 0.9058 - val_auc: 0.9268
    Epoch 18/50
    1280/1280 [==============================] - 2223s 2s/step - loss: 0.2451 - tp0.1: 8257483.8283 - fp0.1: 3209709.3536 - tn0.1: 40707026.2334 - fn0.1: 336440.2756 - precision0.1: 0.7174 - recall0.1: 0.9618 - tp0.3: 7972148.5215 - fp0.3:
     1611985.2998 - tn0.3: 42304753.9649 - fn0.3: 621775.5824 - precision0.3: 0.8295 - recall0.3: 0.9289 - tp0.5: 7592991.5839 - fp0.5: 801375.8977 - tn0.5: 43115384.3333 - fn0.5: 1000932.5199 - precision0.5: 0.9027 - recall0.5: 0.8847 - t
    p0.7: 7121459.2779 - fp0.7: 371705.6339 - tn0.7: 43545020.0328 - fn0.7: 1472464.8259 - precision0.7: 0.9492 - recall0.7: 0.8297 - tp0.9: 6381060.6097 - fp0.9: 111751.2303 - tn0.9: 43804981.0523 - fn0.9: 2212863.4941 - precision0.9: 0.9
    823 - recall0.9: 0.7435 - accuracy: 0.9654 - auc: 0.9764 - val_loss: 0.5303 - val_tp0.1: 3950393.0000 - val_fp0.1: 2825498.0000 - val_tn0.1: 19094474.0000 - val_fn0.1: 344035.0000 - val_precision0.1: 0.5830 - val_recall0.1: 0.9199 - va
    l_tp0.3: 3645339.0000 - val_fp0.3: 2077885.0000 - val_tn0.3: 19842076.0000 - val_fn0.3: 649089.0000 - val_precision0.3: 0.6369 - val_recall0.3: 0.8489 - val_tp0.5: 3283330.0000 - val_fp0.5: 1558246.0000 - val_tn0.5: 20361722.0000 - val
    _fn0.5: 1011098.0000 - val_precision0.5: 0.6782 - val_recall0.5: 0.7646 - val_tp0.7: 2947342.0000 - val_fp0.7: 1176365.0000 - val_tn0.7: 20743604.0000 - val_fn0.7: 1347086.0000 - val_precision0.7: 0.7147 - val_recall0.7: 0.6863 - val_t
    p0.9: 2536959.0000 - val_fp0.9: 698850.0000 - val_tn0.9: 21221120.0000 - val_fn0.9: 1757469.0000 - val_precision0.9: 0.7840 - val_recall0.9: 0.5908 - val_accuracy: 0.9020 - val_auc: 0.9295
    Epoch 19/50
    1280/1280 [==============================] - 2211s 2s/step - loss: 0.2463 - tp0.1: 8262233.3575 - fp0.1: 3162400.5699 - tn0.1: 40752275.6565 - fn0.1: 333748.0101 - precision0.1: 0.7238 - recall0.1: 0.9607 - tp0.3: 7976781.4278 - fp0.3:
     1621499.8923 - tn0.3: 42293181.6620 - fn0.3: 619199.9399 - precision0.3: 0.8316 - recall0.3: 0.9269 - tp0.5: 7600963.2264 - fp0.5: 819732.1842 - tn0.5: 43094936.0523 - fn0.5: 995018.1413 - precision0.5: 0.9028 - recall0.5: 0.8828 - tp
    0.7: 7125748.6019 - fp0.7: 384384.1327 - tn0.7: 43530286.4083 - fn0.7: 1470232.7658 - precision0.7: 0.9489 - recall0.7: 0.8275 - tp0.9: 6355596.0687 - fp0.9: 116562.7260 - tn0.9: 43798125.0718 - fn0.9: 2240385.2990 - precision0.9: 0.98
    20 - recall0.9: 0.7377 - accuracy: 0.9650 - auc: 0.9759 - val_loss: 0.5854 - val_tp0.1: 3832029.0000 - val_fp0.1: 3030240.0000 - val_tn0.1: 18976838.0000 - val_fn0.1: 375292.0000 - val_precision0.1: 0.5584 - val_recall0.1: 0.9108 - val
    _tp0.3: 3499789.0000 - val_fp0.3: 2341510.0000 - val_tn0.3: 19665576.0000 - val_fn0.3: 707532.0000 - val_precision0.3: 0.5991 - val_recall0.3: 0.8318 - val_tp0.5: 3162666.0000 - val_fp0.5: 1856596.0000 - val_tn0.5: 20150478.0000 - val_
    fn0.5: 1044655.0000 - val_precision0.5: 0.6301 - val_recall0.5: 0.7517 - val_tp0.7: 2854187.0000 - val_fp0.7: 1428102.0000 - val_tn0.7: 20578972.0000 - val_fn0.7: 1353134.0000 - val_precision0.7: 0.6665 - val_recall0.7: 0.6784 - val_tp
    0.9: 2488186.0000 - val_fp0.9: 879408.0000 - val_tn0.9: 21127672.0000 - val_fn0.9: 1719135.0000 - val_precision0.9: 0.7389 - val_recall0.9: 0.5914 - val_accuracy: 0.8893 - val_auc: 0.9201
    Epoch 20/50
    1280/1280 [==============================] - 2220s 2s/step - loss: 0.2304 - tp0.1: 8074195.9617 - fp0.1: 2966270.8033 - tn0.1: 41146339.4450 - fn0.1: 323842.2264 - precision0.1: 0.7319 - recall0.1: 0.9616 - tp0.3: 7805955.5699 - fp0.3:
     1533470.4364 - tn0.3: 42579145.3279 - fn0.3: 592082.6183 - precision0.3: 0.8364 - recall0.3: 0.9297 - tp0.5: 7448954.0062 - fp0.5: 779671.7213 - tn0.5: 43332921.9258 - fn0.5: 949084.1819 - precision0.5: 0.9056 - recall0.5: 0.8873 - tp
    0.7: 6997774.1819 - fp0.7: 363044.1397 - tn0.7: 43749575.7775 - fn0.7: 1400264.0062 - precision0.7: 0.9510 - recall0.7: 0.8336 - tp0.9: 6267801.5144 - fp0.9: 107614.1304 - tn0.9: 44004989.1429 - fn0.9: 2130236.6737 - precision0.9: 0.98
    33 - recall0.9: 0.7471 - accuracy: 0.9673 - auc: 0.9771 - val_loss: 0.5968 - val_tp0.1: 3918274.0000 - val_fp0.1: 3250251.0000 - val_tn0.1: 18736068.0000 - val_fn0.1: 309803.0000 - val_precision0.1: 0.5466 - val_recall0.1: 0.9267 - val
    _tp0.3: 3617081.0000 - val_fp0.3: 2559029.0000 - val_tn0.3: 19427288.0000 - val_fn0.3: 610996.0000 - val_precision0.3: 0.5857 - val_recall0.3: 0.8555 - val_tp0.5: 3277656.0000 - val_fp0.5: 2084096.0000 - val_tn0.5: 19902236.0000 - val_
    fn0.5: 950421.0000 - val_precision0.5: 0.6113 - val_recall0.5: 0.7752 - val_tp0.7: 2954727.0000 - val_fp0.7: 1696039.0000 - val_tn0.7: 20290280.0000 - val_fn0.7: 1273350.0000 - val_precision0.7: 0.6353 - val_recall0.7: 0.6988 - val_tp0
    .9: 2580804.0000 - val_fp0.9: 1147305.0000 - val_tn0.9: 20839018.0000 - val_fn0.9: 1647273.0000 - val_precision0.9: 0.6923 - val_recall0.9: 0.6104 - val_accuracy: 0.8842 - val_auc: 0.9226
    Epoch 21/50
    1280/1280 [==============================] - 2204s 2s/step - loss: 0.2384 - tp0.1: 8140617.7596 - fp0.1: 3066144.5098 - tn0.1: 40972600.1975 - fn0.1: 331283.3044 - precision0.1: 0.7253 - recall0.1: 0.9608 - tp0.3: 7870474.5457 - fp0.3:
     1553244.6440 - tn0.3: 42485524.1530 - fn0.3: 601426.5183 - precision0.3: 0.8351 - recall0.3: 0.9289 - tp0.5: 7506928.4785 - fp0.5: 777322.4692 - tn0.5: 43261445.7369 - fn0.5: 964972.5855 - precision0.5: 0.9063 - recall0.5: 0.8856 - tp
    0.7: 7045491.5386 - fp0.7: 357699.5441 - tn0.7: 43681060.9094 - fn0.7: 1426409.5254 - precision0.7: 0.9516 - recall0.7: 0.8307 - tp0.9: 6309047.4247 - fp0.9: 104611.4294 - tn0.9: 43934120.4317 - fn0.9: 2162853.6393 - precision0.9: 0.98
    35 - recall0.9: 0.7429 - accuracy: 0.9668 - auc: 0.9764 - val_loss: 0.5731 - val_tp0.1: 3942096.0000 - val_fp0.1: 2995461.0000 - val_tn0.1: 18887014.0000 - val_fn0.1: 389824.0000 - val_precision0.1: 0.5682 - val_recall0.1: 0.9100 - val
    _tp0.3: 3599028.0000 - val_fp0.3: 2149008.0000 - val_tn0.3: 19733476.0000 - val_fn0.3: 732892.0000 - val_precision0.3: 0.6261 - val_recall0.3: 0.8308 - val_tp0.5: 3252555.0000 - val_fp0.5: 1599632.0000 - val_tn0.5: 20282852.0000 - val_
    fn0.5: 1079365.0000 - val_precision0.5: 0.6703 - val_recall0.5: 0.7508 - val_tp0.7: 2923628.0000 - val_fp0.7: 1191800.0000 - val_tn0.7: 20690680.0000 - val_fn0.7: 1408292.0000 - val_precision0.7: 0.7104 - val_recall0.7: 0.6749 - val_tp
    0.9: 2521604.0000 - val_fp0.9: 728892.0000 - val_tn0.9: 21153588.0000 - val_fn0.9: 1810316.0000 - val_precision0.9: 0.7758 - val_recall0.9: 0.5821 - val_accuracy: 0.8978 - val_auc: 0.9225
    Epoch 22/50
    1280/1280 [==============================] - 2202s 2s/step - loss: 0.2325 - tp0.1: 8142655.7533 - fp0.1: 2988317.6230 - tn0.1: 41059668.6183 - fn0.1: 320016.3560 - precision0.1: 0.7319 - recall0.1: 0.9621 - tp0.3: 7871852.7736 - fp0.3:
     1506270.7799 - tn0.3: 42541714.4848 - fn0.3: 590819.3357 - precision0.3: 0.8406 - recall0.3: 0.9295 - tp0.5: 7524881.3895 - fp0.5: 772793.9836 - tn0.5: 43275195.0945 - fn0.5: 937790.7198 - precision0.5: 0.9075 - recall0.5: 0.8882 - tp
    0.7: 7077438.3216 - fp0.7: 364340.6877 - tn0.7: 43683658.4301 - fn0.7: 1385233.7877 - precision0.7: 0.9513 - recall0.7: 0.8352 - tp0.9: 6339229.0827 - fp0.9: 109004.6729 - tn0.9: 43938973.3130 - fn0.9: 2123443.0265 - precision0.9: 0.98
    30 - recall0.9: 0.7477 - accuracy: 0.9672 - auc: 0.9774 - val_loss: 0.5088 - val_tp0.1: 4052314.0000 - val_fp0.1: 2813624.0000 - val_tn0.1: 18998480.0000 - val_fn0.1: 349977.0000 - val_precision0.1: 0.5902 - val_recall0.1: 0.9205 - val
    _tp0.3: 3687682.0000 - val_fp0.3: 1991405.0000 - val_tn0.3: 19820706.0000 - val_fn0.3: 714609.0000 - val_precision0.3: 0.6493 - val_recall0.3: 0.8377 - val_tp0.5: 3320706.0000 - val_fp0.5: 1380622.0000 - val_tn0.5: 20431490.0000 - val_
    fn0.5: 1081585.0000 - val_precision0.5: 0.7063 - val_recall0.5: 0.7543 - val_tp0.7: 2967625.0000 - val_fp0.7: 953933.0000 - val_tn0.7: 20858176.0000 - val_fn0.7: 1434666.0000 - val_precision0.7: 0.7567 - val_recall0.7: 0.6741 - val_tp0
    .9: 2531957.0000 - val_fp0.9: 497718.0000 - val_tn0.9: 21314390.0000 - val_fn0.9: 1870334.0000 - val_precision0.9: 0.8357 - val_recall0.9: 0.5751 - val_accuracy: 0.9061 - val_auc: 0.9328
    Epoch 23/50
    1280/1280 [==============================] - 2222s 2s/step - loss: 0.2366 - tp0.1: 8347653.8970 - fp0.1: 3081670.7681 - tn0.1: 40757305.4309 - fn0.1: 324014.3521 - precision0.1: 0.7290 - recall0.1: 0.9623 - tp0.3: 8070883.3950 - fp0.3:
     1575587.0781 - tn0.3: 42263394.8119 - fn0.3: 600784.8540 - precision0.3: 0.8352 - recall0.3: 0.9305 - tp0.5: 7717891.5347 - fp0.5: 814813.8595 - tn0.5: 43024150.6456 - fn0.5: 953776.7143 - precision0.5: 0.9036 - recall0.5: 0.8895 - tp
    0.7: 7268612.4356 - fp0.7: 391109.7408 - tn0.7: 43447878.6557 - fn0.7: 1403055.8134 - precision0.7: 0.9483 - recall0.7: 0.8378 - tp0.9: 6489543.3856 - fp0.9: 111622.6698 - tn0.9: 43727379.5621 - fn0.9: 2182124.8634 - precision0.9: 0.98
    31 - recall0.9: 0.7476 - accuracy: 0.9663 - auc: 0.9772 - val_loss: 0.6052 - val_tp0.1: 3963049.0000 - val_fp0.1: 3225080.0000 - val_tn0.1: 18705446.0000 - val_fn0.1: 320823.0000 - val_precision0.1: 0.5513 - val_recall0.1: 0.9251 - val
    _tp0.3: 3674699.0000 - val_fp0.3: 2561181.0000 - val_tn0.3: 19369346.0000 - val_fn0.3: 609173.0000 - val_precision0.3: 0.5893 - val_recall0.3: 0.8578 - val_tp0.5: 3373791.0000 - val_fp0.5: 2127280.0000 - val_tn0.5: 19803252.0000 - val_
    fn0.5: 910081.0000 - val_precision0.5: 0.6133 - val_recall0.5: 0.7876 - val_tp0.7: 3068979.0000 - val_fp0.7: 1731571.0000 - val_tn0.7: 20198960.0000 - val_fn0.7: 1214893.0000 - val_precision0.7: 0.6393 - val_recall0.7: 0.7164 - val_tp0
    .9: 2716240.0000 - val_fp0.9: 1223118.0000 - val_tn0.9: 20707410.0000 - val_fn0.9: 1567632.0000 - val_precision0.9: 0.6895 - val_recall0.9: 0.6341 - val_accuracy: 0.8841 - val_auc: 0.9237
    Epoch 24/50
    1280/1280 [==============================] - 2204s 2s/step - loss: 0.2289 - tp0.1: 8179279.4356 - fp0.1: 3005191.0726 - tn0.1: 41009950.9407 - fn0.1: 316225.4254 - precision0.1: 0.7325 - recall0.1: 0.9631 - tp0.3: 7901240.7315 - fp0.3:
     1504773.7026 - tn0.3: 42510372.7838 - fn0.3: 594264.1296 - precision0.3: 0.8407 - recall0.3: 0.9307 - tp0.5: 7554121.0180 - fp0.5: 777848.4996 - tn0.5: 43237301.8970 - fn0.5: 941383.8431 - precision0.5: 0.9069 - recall0.5: 0.8901 - tp
    0.7: 7115199.6105 - fp0.7: 370807.8002 - tn0.7: 43644327.6066 - fn0.7: 1380305.2506 - precision0.7: 0.9505 - recall0.7: 0.8385 - tp0.9: 6375731.2256 - fp0.9: 109566.7822 - tn0.9: 43905607.9274 - fn0.9: 2119773.6354 - precision0.9: 0.98
    30 - recall0.9: 0.7519 - accuracy: 0.9674 - auc: 0.9780 - val_loss: 0.5914 - val_tp0.1: 4024334.0000 - val_fp0.1: 3404652.0000 - val_tn0.1: 18506274.0000 - val_fn0.1: 279143.0000 - val_precision0.1: 0.5417 - val_recall0.1: 0.9351 - val
    _tp0.3: 3754317.0000 - val_fp0.3: 2715350.0000 - val_tn0.3: 19195568.0000 - val_fn0.3: 549160.0000 - val_precision0.3: 0.5803 - val_recall0.3: 0.8724 - val_tp0.5: 3470133.0000 - val_fp0.5: 2252806.0000 - val_tn0.5: 19658116.0000 - val_
    fn0.5: 833344.0000 - val_precision0.5: 0.6064 - val_recall0.5: 0.8064 - val_tp0.7: 3161773.0000 - val_fp0.7: 1777262.0000 - val_tn0.7: 20133664.0000 - val_fn0.7: 1141704.0000 - val_precision0.7: 0.6402 - val_recall0.7: 0.7347 - val_tp0
    .9: 2787618.0000 - val_fp0.9: 1197991.0000 - val_tn0.9: 20712926.0000 - val_fn0.9: 1515859.0000 - val_precision0.9: 0.6994 - val_recall0.9: 0.6478 - val_accuracy: 0.8823 - val_auc: 0.9283
    Epoch 25/50
    1280/1280 [==============================] - 2226s 2s/step - loss: 0.2252 - tp0.1: 8291979.4137 - fp0.1: 2984606.2982 - tn0.1: 40931178.3692 - fn0.1: 302890.0656 - precision0.1: 0.7344 - recall0.1: 0.9653 - tp0.3: 8019589.6370 - fp0.3:
     1494741.8806 - tn0.3: 42421057.4286 - fn0.3: 575279.8423 - precision0.3: 0.8422 - recall0.3: 0.9335 - tp0.5: 7686923.1866 - fp0.5: 778234.4176 - tn0.5: 43137535.1632 - fn0.5: 907946.2927 - precision0.5: 0.9079 - recall0.5: 0.8947 - tp
    0.7: 7250107.4707 - fp0.7: 372459.3240 - tn0.7: 43543354.1975 - fn0.7: 1344762.0086 - precision0.7: 0.9509 - recall0.7: 0.8437 - tp0.9: 6493233.8290 - fp0.9: 107705.5613 - tn0.9: 43808088.1733 - fn0.9: 2101635.6503 - precision0.9: 0.98
    38 - recall0.9: 0.7548 - accuracy: 0.9677 - auc: 0.9792 - val_loss: 0.5570 - val_tp0.1: 3995650.0000 - val_fp0.1: 3060009.0000 - val_tn0.1: 18836932.0000 - val_fn0.1: 321813.0000 - val_precision0.1: 0.5663 - val_recall0.1: 0.9255 - val
    _tp0.3: 3674821.0000 - val_fp0.3: 2374036.0000 - val_tn0.3: 19522900.0000 - val_fn0.3: 642642.0000 - val_precision0.3: 0.6075 - val_recall0.3: 0.8512 - val_tp0.5: 3346516.0000 - val_fp0.5: 1900974.0000 - val_tn0.5: 19995958.0000 - val_
    fn0.5: 970947.0000 - val_precision0.5: 0.6377 - val_recall0.5: 0.7751 - val_tp0.7: 3018485.0000 - val_fp0.7: 1460209.0000 - val_tn0.7: 20436728.0000 - val_fn0.7: 1298978.0000 - val_precision0.7: 0.6740 - val_recall0.7: 0.6991 - val_tp0
    .9: 2609054.0000 - val_fp0.9: 880392.0000 - val_tn0.9: 21016544.0000 - val_fn0.9: 1708409.0000 - val_precision0.9: 0.7477 - val_recall0.9: 0.6043 - val_accuracy: 0.8904 - val_auc: 0.9280
    Epoch 26/50
    1280/1280 [==============================] - 2216s 2s/step - loss: 0.2287 - tp0.1: 8319398.9969 - fp0.1: 2952438.8306 - tn0.1: 40921029.8501 - fn0.1: 317787.0890 - precision0.1: 0.7394 - recall0.1: 0.9635 - tp0.3: 8049496.8860 - fp0.3:
     1491537.2982 - tn0.3: 42381938.1233 - fn0.3: 587689.1998 - precision0.3: 0.8439 - recall0.3: 0.9326 - tp0.5: 7714917.0734 - fp0.5: 782715.8696 - tn0.5: 43090766.7041 - fn0.5: 922269.0125 - precision0.5: 0.9077 - recall0.5: 0.8941 - tp
    0.7: 7279420.8743 - fp0.7: 371072.6573 - tn0.7: 43502429.9672 - fn0.7: 1357765.2116 - precision0.7: 0.9513 - recall0.7: 0.8438 - tp0.9: 6527109.8852 - fp0.9: 103689.7635 - tn0.9: 43769772.2685 - fn0.9: 2110076.2006 - precision0.9: 0.98
    45 - recall0.9: 0.7564 - accuracy: 0.9674 - auc: 0.9783 - val_loss: 0.5787 - val_tp0.1: 3910190.0000 - val_fp0.1: 3250030.0000 - val_tn0.1: 18794728.0000 - val_fn0.1: 259453.0000 - val_precision0.1: 0.5461 - val_recall0.1: 0.9378 - val
    _tp0.3: 3639624.0000 - val_fp0.3: 2570780.0000 - val_tn0.3: 19473980.0000 - val_fn0.3: 530019.0000 - val_precision0.3: 0.5861 - val_recall0.3: 0.8729 - val_tp0.5: 3361012.0000 - val_fp0.5: 2146148.0000 - val_tn0.5: 19898600.0000 - val_
    fn0.5: 808631.0000 - val_precision0.5: 0.6103 - val_recall0.5: 0.8061 - val_tp0.7: 3051142.0000 - val_fp0.7: 1790049.0000 - val_tn0.7: 20254708.0000 - val_fn0.7: 1118501.0000 - val_precision0.7: 0.6302 - val_recall0.7: 0.7318 - val_tp0
    .9: 2683794.0000 - val_fp0.9: 1283960.0000 - val_tn0.9: 20760796.0000 - val_fn0.9: 1485849.0000 - val_precision0.9: 0.6764 - val_recall0.9: 0.6437 - val_accuracy: 0.8873 - val_auc: 0.9297
    Epoch 27/50
    1280/1280 [==============================] - 2202s 2s/step - loss: 0.2301 - tp0.1: 8107973.9399 - fp0.1: 2902140.6105 - tn0.1: 41182030.2319 - fn0.1: 318498.9274 - precision0.1: 0.7370 - recall0.1: 0.9620 - tp0.3: 7841754.6401 - fp0.3:
     1478702.1952 - tn0.3: 42605477.3365 - fn0.3: 584718.2272 - precision0.3: 0.8414 - recall0.3: 0.9300 - tp0.5: 7511893.1749 - fp0.5: 775511.8610 - tn0.5: 43308683.9024 - fn0.5: 914579.6924 - precision0.5: 0.9061 - recall0.5: 0.8905 - tp
    0.7: 7075173.0062 - fp0.7: 368300.9977 - tn0.7: 43715881.6714 - fn0.7: 1351299.8610 - precision0.7: 0.9500 - recall0.7: 0.8385 - tp0.9: 6326135.5004 - fp0.9: 105903.6909 - tn0.9: 43978285.1374 - fn0.9: 2100337.3669 - precision0.9: 0.98
    29 - recall0.9: 0.7495 - accuracy: 0.9673 - auc: 0.9774 - val_loss: 0.5563 - val_tp0.1: 3973953.0000 - val_fp0.1: 3033629.0000 - val_tn0.1: 18923208.0000 - val_fn0.1: 283603.0000 - val_precision0.1: 0.5671 - val_recall0.1: 0.9334 - val
    _tp0.3: 3702270.0000 - val_fp0.3: 2375708.0000 - val_tn0.3: 19581140.0000 - val_fn0.3: 555286.0000 - val_precision0.3: 0.6091 - val_recall0.3: 0.8696 - val_tp0.5: 3427641.0000 - val_fp0.5: 1972155.0000 - val_tn0.5: 19984684.0000 - val_
    fn0.5: 829915.0000 - val_precision0.5: 0.6348 - val_recall0.5: 0.8051 - val_tp0.7: 3124472.0000 - val_fp0.7: 1581643.0000 - val_tn0.7: 20375196.0000 - val_fn0.7: 1133084.0000 - val_precision0.7: 0.6639 - val_recall0.7: 0.7339 - val_tp0
    .9: 2757974.0000 - val_fp0.9: 1109127.0000 - val_tn0.9: 20847720.0000 - val_fn0.9: 1499582.0000 - val_precision0.9: 0.7132 - val_recall0.9: 0.6478 - val_accuracy: 0.8931 - val_auc: 0.9321
    Epoch 28/50
    1280/1280 [==============================] - 2087s 2s/step - loss: 0.2197 - tp0.1: 8160795.8228 - fp0.1: 2901034.4309 - tn0.1: 41142814.8181 - fn0.1: 306021.3466 - precision0.1: 0.7381 - recall0.1: 0.9644 - tp0.3: 7892365.4137 - fp0.3:
     1440828.1148 - tn0.3: 42603028.9719 - fn0.3: 574451.7557 - precision0.3: 0.8461 - recall0.3: 0.9326 - tp0.5: 7566706.5480 - fp0.5: 755226.4699 - tn0.5: 43288602.1678 - fn0.5: 900110.6214 - precision0.5: 0.9096 - recall0.5: 0.8942 - tp
    0.7: 7138328.0726 - fp0.7: 359169.3607 - tn0.7: 43684677.2763 - fn0.7: 1328489.0968 - precision0.7: 0.9523 - recall0.7: 0.8435 - tp0.9: 6407064.9711 - fp0.9: 100987.8735 - tn0.9: 43942848.2186 - fn0.9: 2059752.1983 - precision0.9: 0.98
    47 - recall0.9: 0.7566 - accuracy: 0.9687 - auc: 0.9791 - val_loss: 0.4500 - val_tp0.1: 3927337.0000 - val_fp0.1: 2709904.0000 - val_tn0.1: 19279502.0000 - val_fn0.1: 297660.0000 - val_precision0.1: 0.5917 - val_recall0.1: 0.9295 - val
    _tp0.3: 3622327.0000 - val_fp0.3: 1934449.0000 - val_tn0.3: 20054960.0000 - val_fn0.3: 602670.0000 - val_precision0.3: 0.6519 - val_recall0.3: 0.8574 - val_tp0.5: 3304686.0000 - val_fp0.5: 1267577.0000 - val_tn0.5: 20721824.0000 - val_
    fn0.5: 920311.0000 - val_precision0.5: 0.7228 - val_recall0.5: 0.7822 - val_tp0.7: 2969971.0000 - val_fp0.7: 803372.0000 - val_tn0.7: 21186032.0000 - val_fn0.7: 1255026.0000 - val_precision0.7: 0.7871 - val_recall0.7: 0.7030 - val_tp0.
    9: 2545450.0000 - val_fp0.9: 373527.0000 - val_tn0.9: 21615866.0000 - val_fn0.9: 1679547.0000 - val_precision0.9: 0.8720 - val_recall0.9: 0.6025 - val_accuracy: 0.9165 - val_auc: 0.9426
    Epoch 29/50
    1280/1280 [==============================] - 1968s 2s/step - loss: 0.2223 - tp0.1: 8375369.2490 - fp0.1: 2828068.0578 - tn0.1: 41000432.9828 - fn0.1: 306798.1413 - precision0.1: 0.7500 - recall0.1: 0.9648 - tp0.3: 8117816.3411 - fp0.3:
     1454670.2927 - tn0.3: 42373821.8306 - fn0.3: 564351.0492 - precision0.3: 0.8489 - recall0.3: 0.9354 - tp0.5: 7795654.3497 - fp0.5: 777398.3349 - tn0.5: 43051103.4832 - fn0.5: 886513.0406 - precision0.5: 0.9098 - recall0.5: 0.8987 - tp
    0.7: 7368227.7635 - fp0.7: 376034.1069 - tn0.7: 43452466.5371 - fn0.7: 1313939.6269 - precision0.7: 0.9517 - recall0.7: 0.8498 - tp0.9: 6617754.3536 - fp0.9: 111597.3193 - tn0.9: 43716853.5761 - fn0.9: 2064413.0367 - precision0.9: 0.98
    35 - recall0.9: 0.7637 - accuracy: 0.9681 - auc: 0.9792 - val_loss: 0.5416 - val_tp0.1: 3960357.0000 - val_fp0.1: 3230144.0000 - val_tn0.1: 18775996.0000 - val_fn0.1: 247899.0000 - val_precision0.1: 0.5508 - val_recall0.1: 0.9411 - val
    _tp0.3: 3709415.0000 - val_fp0.3: 2490342.0000 - val_tn0.3: 19515800.0000 - val_fn0.3: 498841.0000 - val_precision0.3: 0.5983 - val_recall0.3: 0.8815 - val_tp0.5: 3451033.0000 - val_fp0.5: 2038241.0000 - val_tn0.5: 19967904.0000 - val_
    fn0.5: 757223.0000 - val_precision0.5: 0.6287 - val_recall0.5: 0.8201 - val_tp0.7: 3148179.0000 - val_fp0.7: 1621294.0000 - val_tn0.7: 20384850.0000 - val_fn0.7: 1060077.0000 - val_precision0.7: 0.6601 - val_recall0.7: 0.7481 - val_tp0
    .9: 2788544.0000 - val_fp0.9: 1101598.0000 - val_tn0.9: 20904542.0000 - val_fn0.9: 1419712.0000 - val_precision0.9: 0.7168 - val_recall0.9: 0.6626 - val_accuracy: 0.8934 - val_auc: 0.9369
    Epoch 30/50
    1280/1280 [==============================] - 1987s 2s/step - loss: 0.2171 - tp0.1: 8237924.9360 - fp0.1: 2817507.8111 - tn0.1: 41152156.1608 - fn0.1: 303072.4856 - precision0.1: 0.7480 - recall0.1: 0.9644 - tp0.3: 7981439.7697 - fp0.3:
     1425495.2022 - tn0.3: 42544167.9969 - fn0.3: 559557.6518 - precision0.3: 0.8500 - recall0.3: 0.9347 - tp0.5: 7663120.4309 - fp0.5: 752759.7705 - tn0.5: 43216900.0492 - fn0.5: 877876.9906 - precision0.5: 0.9116 - recall0.5: 0.8978 - tp
    0.7: 7238382.6721 - fp0.7: 357392.6354 - tn0.7: 43612263.4215 - fn0.7: 1302614.7494 - precision0.7: 0.9538 - recall0.7: 0.8481 - tp0.9: 6495143.9938 - fp0.9: 101203.1015 - tn0.9: 43868469.8891 - fn0.9: 2045853.4278 - precision0.9: 0.98
    52 - recall0.9: 0.7605 - accuracy: 0.9692 - auc: 0.9793 - val_loss: 0.5477 - val_tp0.1: 4026668.0000 - val_fp0.1: 3118480.0000 - val_tn0.1: 18786898.0000 - val_fn0.1: 282350.0000 - val_precision0.1: 0.5636 - val_recall0.1: 0.9345 - val
    _tp0.3: 3769431.0000 - val_fp0.3: 2411082.0000 - val_tn0.3: 19494304.0000 - val_fn0.3: 539587.0000 - val_precision0.3: 0.6099 - val_recall0.3: 0.8748 - val_tp0.5: 3496580.0000 - val_fp0.5: 1951231.0000 - val_tn0.5: 19954152.0000 - val_
    fn0.5: 812438.0000 - val_precision0.5: 0.6418 - val_recall0.5: 0.8115 - val_tp0.7: 3175729.0000 - val_fp0.7: 1538001.0000 - val_tn0.7: 20367384.0000 - val_fn0.7: 1133289.0000 - val_precision0.7: 0.6737 - val_recall0.7: 0.7370 - val_tp0
    .9: 2769656.0000 - val_fp0.9: 1014413.0000 - val_tn0.9: 20890968.0000 - val_fn0.9: 1539362.0000 - val_precision0.9: 0.7319 - val_recall0.9: 0.6428 - val_accuracy: 0.8946 - val_auc: 0.9332
    Epoch 31/50
    1280/1280 [==============================] - 1975s 2s/step - loss: 0.2140 - tp0.1: 8254893.9524 - fp0.1: 2820541.8767 - tn0.1: 41143836.3778 - fn0.1: 291401.7947 - precision0.1: 0.7449 - recall0.1: 0.9663 - tp0.3: 8001833.3880 - fp0.3:
     1415982.6237 - tn0.3: 42548340.3044 - fn0.3: 544462.3591 - precision0.3: 0.8490 - recall0.3: 0.9370 - tp0.5: 7689740.5105 - fp0.5: 749876.5176 - tn0.5: 43214484.3458 - fn0.5: 856555.2365 - precision0.5: 0.9104 - recall0.5: 0.9008 - tp
    0.7: 7273005.2974 - fp0.7: 357730.3216 - tn0.7: 43606618.3302 - fn0.7: 1273290.4496 - precision0.7: 0.9525 - recall0.7: 0.8527 - tp0.9: 6528857.1772 - fp0.9: 101215.2264 - tn0.9: 43863149.4848 - fn0.9: 2017438.5699 - precision0.9: 0.98
    45 - recall0.9: 0.7661 - accuracy: 0.9694 - auc: 0.9801 - val_loss: 0.6223 - val_tp0.1: 3930460.0000 - val_fp0.1: 3394464.0000 - val_tn0.1: 18612148.0000 - val_fn0.1: 277327.0000 - val_precision0.1: 0.5366 - val_recall0.1: 0.9341 - val
    _tp0.3: 3651274.0000 - val_fp0.3: 2652264.0000 - val_tn0.3: 19354352.0000 - val_fn0.3: 556513.0000 - val_precision0.3: 0.5792 - val_recall0.3: 0.8677 - val_tp0.5: 3363770.0000 - val_fp0.5: 2181393.0000 - val_tn0.5: 19825220.0000 - val_
    fn0.5: 844017.0000 - val_precision0.5: 0.6066 - val_recall0.5: 0.7994 - val_tp0.7: 3042185.0000 - val_fp0.7: 1805943.0000 - val_tn0.7: 20200670.0000 - val_fn0.7: 1165602.0000 - val_precision0.7: 0.6275 - val_recall0.7: 0.7230 - val_tp0
    .9: 2649971.0000 - val_fp0.9: 1376978.0000 - val_tn0.9: 20629630.0000 - val_fn0.9: 1557816.0000 - val_precision0.9: 0.6581 - val_recall0.9: 0.6298 - val_accuracy: 0.8846 - val_auc: 0.9230
    Epoch 32/50
    1280/1280 [==============================] - 1985s 2s/step - loss: 0.2162 - tp0.1: 8168329.4106 - fp0.1: 2823874.2186 - tn0.1: 41219292.6159 - fn0.1: 299147.5254 - precision0.1: 0.7416 - recall0.1: 0.9648 - tp0.3: 7902494.5543 - fp0.3:
     1406775.2030 - tn0.3: 42636384.1311 - fn0.3: 564982.3817 - precision0.3: 0.8486 - recall0.3: 0.9329 - tp0.5: 7587653.1304 - fp0.5: 746883.7400 - tn0.5: 43296303.4005 - fn0.5: 879823.8056 - precision0.5: 0.9100 - recall0.5: 0.8957 - tp
    0.7: 7172023.5183 - fp0.7: 358490.8813 - tn0.7: 43684701.1390 - fn0.7: 1295453.4176 - precision0.7: 0.9519 - recall0.7: 0.8466 - tp0.9: 6440127.4231 - fp0.9: 99619.9977 - tn0.9: 43943567.7978 - fn0.9: 2027349.5129 - precision0.9: 0.984
    4 - recall0.9: 0.7605 - accuracy: 0.9691 - auc: 0.9792 - val_loss: 0.5922 - val_tp0.1: 4021661.0000 - val_fp0.1: 3341968.0000 - val_tn0.1: 18581152.0000 - val_fn0.1: 269622.0000 - val_precision0.1: 0.5462 - val_recall0.1: 0.9372 - val_
    tp0.3: 3723109.0000 - val_fp0.3: 2641493.0000 - val_tn0.3: 19281632.0000 - val_fn0.3: 568174.0000 - val_precision0.3: 0.5850 - val_recall0.3: 0.8676 - val_tp0.5: 3417861.0000 - val_fp0.5: 2185964.0000 - val_tn0.5: 19737154.0000 - val_f
    n0.5: 873422.0000 - val_precision0.5: 0.6099 - val_recall0.5: 0.7965 - val_tp0.7: 3082813.0000 - val_fp0.7: 1779173.0000 - val_tn0.7: 20143952.0000 - val_fn0.7: 1208470.0000 - val_precision0.7: 0.6341 - val_recall0.7: 0.7184 - val_tp0.
    9: 2687718.0000 - val_fp0.9: 1223941.0000 - val_tn0.9: 20699176.0000 - val_fn0.9: 1603565.0000 - val_precision0.9: 0.6871 - val_recall0.9: 0.6263 - val_accuracy: 0.8833 - val_auc: 0.9267
    Epoch 33/50
    1280/1280 [==============================] - 1990s 2s/step - loss: 0.2153 - tp0.1: 8159566.2951 - fp0.1: 2769324.2404 - tn0.1: 41293176.3880 - fn0.1: 288608.7385 - precision0.1: 0.7434 - recall0.1: 0.9660 - tp0.3: 7912034.2069 - fp0.3:
     1418334.3794 - tn0.3: 42644140.5886 - fn0.3: 536140.8267 - precision0.3: 0.8451 - recall0.3: 0.9365 - tp0.5: 7606237.6534 - fp0.5: 760974.3443 - tn0.5: 43301518.3575 - fn0.5: 841937.3802 - precision0.5: 0.9067 - recall0.5: 0.9000 - tp
    0.7: 7188970.2584 - fp0.7: 368072.7486 - tn0.7: 43694407.5129 - fn0.7: 1259204.7752 - precision0.7: 0.9494 - recall0.7: 0.8501 - tp0.9: 6447265.7268 - fp0.9: 106096.6183 - tn0.9: 43956391.5020 - fn0.9: 2000909.3068 - precision0.9: 0.98
    25 - recall0.9: 0.7622 - accuracy: 0.9690 - auc: 0.9797 - val_loss: 0.6103 - val_tp0.1: 4142693.0000 - val_fp0.1: 3021502.0000 - val_tn0.1: 18751128.0000 - val_fn0.1: 299079.0000 - val_precision0.1: 0.5782 - val_recall0.1: 0.9327 - val
    _tp0.3: 3873213.0000 - val_fp0.3: 2448264.0000 - val_tn0.3: 19324360.0000 - val_fn0.3: 568559.0000 - val_precision0.3: 0.6127 - val_recall0.3: 0.8720 - val_tp0.5: 3610020.0000 - val_fp0.5: 2106610.0000 - val_tn0.5: 19666014.0000 - val_
    fn0.5: 831752.0000 - val_precision0.5: 0.6315 - val_recall0.5: 0.8127 - val_tp0.7: 3302152.0000 - val_fp0.7: 1796893.0000 - val_tn0.7: 19975734.0000 - val_fn0.7: 1139620.0000 - val_precision0.7: 0.6476 - val_recall0.7: 0.7434 - val_tp0
    .9: 2940645.0000 - val_fp0.9: 1387374.0000 - val_tn0.9: 20385260.0000 - val_fn0.9: 1501127.0000 - val_precision0.9: 0.6794 - val_recall0.9: 0.6620 - val_accuracy: 0.8879 - val_auc: 0.9267
    Epoch 34/50
    1280/1280 [==============================] - 1975s 2s/step - loss: 0.2140 - tp0.1: 8354952.6027 - fp0.1: 2805323.7728 - tn0.1: 41057643.9641 - fn0.1: 292731.8275 - precision0.1: 0.7496 - recall0.1: 0.9661 - tp0.3: 8100316.4520 - fp0.3:
     1408795.9087 - tn0.3: 42454187.0320 - fn0.3: 547367.9781 - precision0.3: 0.8514 - recall0.3: 0.9370 - tp0.5: 7794247.9141 - fp0.5: 754289.6760 - tn0.5: 43108644.9532 - fn0.5: 853436.5160 - precision0.5: 0.9115 - recall0.5: 0.9017 - tp
    0.7: 7381753.1905 - fp0.7: 366912.4387 - tn0.7: 43496065.3107 - fn0.7: 1265931.2397 - precision0.7: 0.9524 - recall0.7: 0.8539 - tp0.9: 6643385.3435 - fp0.9: 104367.8337 - tn0.9: 43758580.2849 - fn0.9: 2004299.0867 - precision0.9: 0.98
    45 - recall0.9: 0.7683 - accuracy: 0.9694 - auc: 0.9801 - val_loss: 0.6624 - val_tp0.1: 3856354.0000 - val_fp0.1: 3145321.0000 - val_tn0.1: 18842388.0000 - val_fn0.1: 370344.0000 - val_precision0.1: 0.5508 - val_recall0.1: 0.9124 - val
    _tp0.3: 3549372.0000 - val_fp0.3: 2542341.0000 - val_tn0.3: 19445350.0000 - val_fn0.3: 677326.0000 - val_precision0.3: 0.5827 - val_recall0.3: 0.8398 - val_tp0.5: 3241294.0000 - val_fp0.5: 2119984.0000 - val_tn0.5: 19867720.0000 - val_
    fn0.5: 985404.0000 - val_precision0.5: 0.6046 - val_recall0.5: 0.7669 - val_tp0.7: 2930029.0000 - val_fp0.7: 1781338.0000 - val_tn0.7: 20206368.0000 - val_fn0.7: 1296669.0000 - val_precision0.7: 0.6219 - val_recall0.7: 0.6932 - val_tp0
    .9: 2550993.0000 - val_fp0.9: 1347073.0000 - val_tn0.9: 20640632.0000 - val_fn0.9: 1675705.0000 - val_precision0.9: 0.6544 - val_recall0.9: 0.6035 - val_accuracy: 0.8815 - val_auc: 0.9118
    Epoch 35/50
    1280/1280 [==============================] - 1979s 2s/step - loss: 0.2136 - tp0.1: 8121225.3489 - fp0.1: 2767519.3271 - tn0.1: 41328977.4340 - fn0.1: 292944.1897 - precision0.1: 0.7441 - recall0.1: 0.9648 - tp0.3: 7867775.7931 - fp0.3:
     1380664.7354 - tn0.3: 42715785.1093 - fn0.3: 546393.7455 - precision0.3: 0.8496 - recall0.3: 0.9344 - tp0.5: 7568656.2834 - fp0.5: 741064.5824 - tn0.5: 43355403.8126 - fn0.5: 845513.2553 - precision0.5: 0.9099 - recall0.5: 0.8989 - tp
    0.7: 7163273.2842 - fp0.7: 355917.4442 - tn0.7: 43740534.9688 - fn0.7: 1250896.2545 - precision0.7: 0.9521 - recall0.7: 0.8507 - tp0.9: 6427745.3263 - fp0.9: 99399.9547 - tn0.9: 43997102.2342 - fn0.9: 1986424.2123 - precision0.9: 0.984
    3 - recall0.9: 0.7629 - accuracy: 0.9697 - auc: 0.9794 - val_loss: 0.6109 - val_tp0.1: 3805391.0000 - val_fp0.1: 3155702.0000 - val_tn0.1: 18859740.0000 - val_fn0.1: 393569.0000 - val_precision0.1: 0.5467 - val_recall0.1: 0.9063 - val_
    tp0.3: 3505616.0000 - val_fp0.3: 2482634.0000 - val_tn0.3: 19532808.0000 - val_fn0.3: 693344.0000 - val_precision0.3: 0.5854 - val_recall0.3: 0.8349 - val_tp0.5: 3205095.0000 - val_fp0.5: 1968560.0000 - val_tn0.5: 20046874.0000 - val_f
    n0.5: 993865.0000 - val_precision0.5: 0.6195 - val_recall0.5: 0.7633 - val_tp0.7: 2889262.0000 - val_fp0.7: 1535981.0000 - val_tn0.7: 20479460.0000 - val_fn0.7: 1309698.0000 - val_precision0.7: 0.6529 - val_recall0.7: 0.6881 - val_tp0.
    9: 2477042.0000 - val_fp0.9: 982485.0000 - val_tn0.9: 21032952.0000 - val_fn0.9: 1721918.0000 - val_precision0.9: 0.7160 - val_recall0.9: 0.5899 - val_accuracy: 0.8870 - val_auc: 0.9160
    Epoch 36/50
    1280/1280 [==============================] - 1979s 2s/step - loss: 0.2059 - tp0.1: 8126280.0625 - fp0.1: 2646687.9227 - tn0.1: 41454888.0211 - fn0.1: 282807.8095 - precision0.1: 0.7538 - recall0.1: 0.9656 - tp0.3: 7878883.7135 - fp0.3:
     1323333.3552 - tn0.3: 42778233.2834 - fn0.3: 530204.1585 - precision0.3: 0.8559 - recall0.3: 0.9362 - tp0.5: 7591944.3318 - fp0.5: 712271.5222 - tn0.5: 43389291.7783 - fn0.5: 817143.5402 - precision0.5: 0.9140 - recall0.5: 0.9022 - tp
    0.7: 7203686.0226 - fp0.7: 343437.4106 - tn0.7: 43758143.1655 - fn0.7: 1205401.8493 - precision0.7: 0.9542 - recall0.7: 0.8564 - tp0.9: 6507343.4036 - fp0.9: 97816.8470 - tn0.9: 44003797.8345 - fn0.9: 1901744.4684 - precision0.9: 0.985
    0 - recall0.9: 0.7738 - accuracy: 0.9709 - auc: 0.9801 - val_loss: 0.5870 - val_tp0.1: 4142690.0000 - val_fp0.1: 3187548.0000 - val_tn0.1: 18594374.0000 - val_fn0.1: 289788.0000 - val_precision0.1: 0.5652 - val_recall0.1: 0.9346 - val_
    tp0.3: 3873916.0000 - val_fp0.3: 2590674.0000 - val_tn0.3: 19191250.0000 - val_fn0.3: 558562.0000 - val_precision0.3: 0.5993 - val_recall0.3: 0.8740 - val_tp0.5: 3589951.0000 - val_fp0.5: 2186008.0000 - val_tn0.5: 19595906.0000 - val_f
    n0.5: 842527.0000 - val_precision0.5: 0.6215 - val_recall0.5: 0.8099 - val_tp0.7: 3248583.0000 - val_fp0.7: 1758534.0000 - val_tn0.7: 20023388.0000 - val_fn0.7: 1183895.0000 - val_precision0.7: 0.6488 - val_recall0.7: 0.7329 - val_tp0.
    9: 2813690.0000 - val_fp0.9: 1197675.0000 - val_tn0.9: 20584246.0000 - val_fn0.9: 1618788.0000 - val_precision0.9: 0.7014 - val_recall0.9: 0.6348 - val_accuracy: 0.8845 - val_auc: 0.9275
    Epoch 37/50
    1280/1280 [==============================] - 1993s 2s/step - loss: 0.2130 - tp0.1: 8244281.2553 - fp0.1: 2777623.1632 - tn0.1: 41197305.6495 - fn0.1: 291461.3638 - precision0.1: 0.7465 - recall0.1: 0.9665 - tp0.3: 7992234.6581 - fp0.3:
     1376180.9282 - tn0.3: 42598749.7158 - fn0.3: 543507.9610 - precision0.3: 0.8514 - recall0.3: 0.9375 - tp0.5: 7696124.9321 - fp0.5: 737935.7494 - tn0.5: 43236985.3677 - fn0.5: 839617.6870 - precision0.5: 0.9113 - recall0.5: 0.9031 - tp
    0.7: 7289915.6175 - fp0.7: 354145.8751 - tn0.7: 43620781.5706 - fn0.7: 1245827.0016 - precision0.7: 0.9529 - recall0.7: 0.8559 - tp0.9: 6568139.6253 - fp0.9: 99108.1030 - tn0.9: 43875848.4980 - fn0.9: 1967602.9938 - precision0.9: 0.985
    0 - recall0.9: 0.7707 - accuracy: 0.9697 - auc: 0.9803 - val_loss: 0.6359 - val_tp0.1: 3972718.0000 - val_fp0.1: 2970398.0000 - val_tn0.1: 18895362.0000 - val_fn0.1: 375917.0000 - val_precision0.1: 0.5722 - val_recall0.1: 0.9136 - val_
    tp0.3: 3696968.0000 - val_fp0.3: 2407542.0000 - val_tn0.3: 19458224.0000 - val_fn0.3: 651667.0000 - val_precision0.3: 0.6056 - val_recall0.3: 0.8501 - val_tp0.5: 3418831.0000 - val_fp0.5: 2053387.0000 - val_tn0.5: 19812380.0000 - val_f
    n0.5: 929804.0000 - val_precision0.5: 0.6248 - val_recall0.5: 0.7862 - val_tp0.7: 3101481.0000 - val_fp0.7: 1742891.0000 - val_tn0.7: 20122868.0000 - val_fn0.7: 1247154.0000 - val_precision0.7: 0.6402 - val_recall0.7: 0.7132 - val_tp0.
    9: 2727270.0000 - val_fp0.9: 1316358.0000 - val_tn0.9: 20549408.0000 - val_fn0.9: 1621365.0000 - val_precision0.9: 0.6745 - val_recall0.9: 0.6272 - val_accuracy: 0.8862 - val_auc: 0.9173
    Epoch 38/50
    1280/1280 [==============================] - 1988s 2s/step - loss: 0.2095 - tp0.1: 8322547.4973 - fp0.1: 2736467.1725 - tn0.1: 41169366.0554 - fn0.1: 282238.6792 - precision0.1: 0.7515 - recall0.1: 0.9670 - tp0.3: 8066398.6003 - fp0.3:
     1365148.2873 - tn0.3: 42540740.5316 - fn0.3: 538387.5761 - precision0.3: 0.8549 - recall0.3: 0.9365 - tp0.5: 7770556.2529 - fp0.5: 735978.7931 - tn0.5: 43169885.3575 - fn0.5: 834229.9235 - precision0.5: 0.9132 - recall0.5: 0.9022 - tp
    0.7: 7368501.9875 - fp0.7: 354363.2201 - tn0.7: 43551529.7572 - fn0.7: 1236284.1889 - precision0.7: 0.9541 - recall0.7: 0.8557 - tp0.9: 6640326.0984 - fp0.9: 97963.4192 - tn0.9: 43807909.1975 - fn0.9: 1964460.0781 - precision0.9: 0.985
    5 - recall0.9: 0.7709 - accuracy: 0.9698 - auc: 0.9806 - val_loss: 0.5532 - val_tp0.1: 4070077.0000 - val_fp0.1: 3091837.0000 - val_tn0.1: 18834880.0000 - val_fn0.1: 217612.0000 - val_precision0.1: 0.5683 - val_recall0.1: 0.9492 - val_
    tp0.3: 3841354.0000 - val_fp0.3: 2343330.0000 - val_tn0.3: 19583380.0000 - val_fn0.3: 446335.0000 - val_precision0.3: 0.6211 - val_recall0.3: 0.8959 - val_tp0.5: 3610309.0000 - val_fp0.5: 1940937.0000 - val_tn0.5: 19985782.0000 - val_f
    n0.5: 677380.0000 - val_precision0.5: 0.6504 - val_recall0.5: 0.8420 - val_tp0.7: 3328203.0000 - val_fp0.7: 1620408.0000 - val_tn0.7: 20306288.0000 - val_fn0.7: 959486.0000 - val_precision0.7: 0.6726 - val_recall0.7: 0.7762 - val_tp0.9
    : 2957699.0000 - val_fp0.9: 1252356.0000 - val_tn0.9: 20674356.0000 - val_fn0.9: 1329990.0000 - val_precision0.9: 0.7025 - val_recall0.9: 0.6898 - val_accuracy: 0.9001 - val_auc: 0.9387
    Epoch 39/50
    1280/1280 [==============================] - 1983s 2s/step - loss: 0.2144 - tp0.1: 8366575.3263 - fp0.1: 2766167.9742 - tn0.1: 41080636.4215 - fn0.1: 297288.0242 - precision0.1: 0.7524 - recall0.1: 0.9656 - tp0.3: 8113164.2248 - fp0.3:
     1368911.6565 - tn0.3: 42477877.7752 - fn0.3: 550699.1257 - precision0.3: 0.8555 - recall0.3: 0.9368 - tp0.5: 7813941.1093 - fp0.5: 738235.6479 - tn0.5: 43108568.5925 - fn0.5: 849922.2412 - precision0.5: 0.9133 - recall0.5: 0.9025 - tp
    0.7: 7408230.3068 - fp0.7: 358148.5066 - tn0.7: 43488651.3341 - fn0.7: 1255633.0437 - precision0.7: 0.9536 - recall0.7: 0.8560 - tp0.9: 6668979.4731 - fp0.9: 100667.4668 - tn0.9: 43746134.9883 - fn0.9: 1994883.8774 - precision0.9: 0.98
    51 - recall0.9: 0.7710 - accuracy: 0.9697 - auc: 0.9798 - val_loss: 0.5509 - val_tp0.1: 3928121.0000 - val_fp0.1: 3362312.0000 - val_tn0.1: 18683104.0000 - val_fn0.1: 240862.0000 - val_precision0.1: 0.5388 - val_recall0.1: 0.9422 - val
    _tp0.3: 3683700.0000 - val_fp0.3: 2630754.0000 - val_tn0.3: 19414668.0000 - val_fn0.3: 485283.0000 - val_precision0.3: 0.5834 - val_recall0.3: 0.8836 - val_tp0.5: 3409649.0000 - val_fp0.5: 2123677.0000 - val_tn0.5: 19921740.0000 - val_
    fn0.5: 759334.0000 - val_precision0.5: 0.6162 - val_recall0.5: 0.8179 - val_tp0.7: 3082747.0000 - val_fp0.7: 1673717.0000 - val_tn0.7: 20371692.0000 - val_fn0.7: 1086236.0000 - val_precision0.7: 0.6481 - val_recall0.7: 0.7394 - val_tp0
    .9: 2675367.0000 - val_fp0.9: 1106165.0000 - val_tn0.9: 20939250.0000 - val_fn0.9: 1493616.0000 - val_precision0.9: 0.7075 - val_recall0.9: 0.6417 - val_accuracy: 0.8900 - val_auc: 0.9336
    Epoch 40/50
    1280/1280 [==============================] - 1983s 2s/step - loss: 0.1969 - tp0.1: 8112039.9274 - fp0.1: 2648956.8977 - tn0.1: 41481438.0515 - fn0.1: 268231.5964 - precision0.1: 0.7550 - recall0.1: 0.9682 - tp0.3: 7868925.6425 - fp0.3:
     1315040.1202 - tn0.3: 42815365.2927 - fn0.3: 511345.8813 - precision0.3: 0.8571 - recall0.3: 0.9396 - tp0.5: 7584341.2420 - fp0.5: 709881.4247 - tn0.5: 43420518.9040 - fn0.5: 795930.2818 - precision0.5: 0.9147 - recall0.5: 0.9057 - tp
    0.7: 7193185.0031 - fp0.7: 342578.8923 - tn0.7: 43787824.5191 - fn0.7: 1187086.5207 - precision0.7: 0.9550 - recall0.7: 0.8587 - tp0.9: 6494738.0343 - fp0.9: 97088.4145 - tn0.9: 44033315.7065 - fn0.9: 1885533.4895 - precision0.9: 0.985
    6 - recall0.9: 0.7753 - accuracy: 0.9718 - auc: 0.9815 - val_loss: 0.5690 - val_tp0.1: 3846468.0000 - val_fp0.1: 3007369.0000 - val_tn0.1: 19101806.0000 - val_fn0.1: 258769.0000 - val_precision0.1: 0.5612 - val_recall0.1: 0.9370 - val_
    tp0.3: 3613817.0000 - val_fp0.3: 2397217.0000 - val_tn0.3: 19711954.0000 - val_fn0.3: 491420.0000 - val_precision0.3: 0.6012 - val_recall0.3: 0.8803 - val_tp0.5: 3367535.0000 - val_fp0.5: 2008822.0000 - val_tn0.5: 20100340.0000 - val_f
    n0.5: 737702.0000 - val_precision0.5: 0.6264 - val_recall0.5: 0.8203 - val_tp0.7: 3076985.0000 - val_fp0.7: 1689883.0000 - val_tn0.7: 20419284.0000 - val_fn0.7: 1028252.0000 - val_precision0.7: 0.6455 - val_recall0.7: 0.7495 - val_tp0.
    9: 2719452.0000 - val_fp0.9: 1280160.0000 - val_tn0.9: 20829004.0000 - val_fn0.9: 1385785.0000 - val_precision0.9: 0.6799 - val_recall0.9: 0.6624 - val_accuracy: 0.8952 - val_auc: 0.9311
    Epoch 41/50
    1280/1280 [==============================] - 1978s 2s/step - loss: 0.2044 - tp0.1: 8119761.2217 - fp0.1: 2626596.4067 - tn0.1: 41483268.4879 - fn0.1: 281035.5894 - precision0.1: 0.7571 - recall0.1: 0.9663 - tp0.3: 7878864.4973 - fp0.3:
     1320586.7525 - tn0.3: 42789243.0968 - fn0.3: 521932.3138 - precision0.3: 0.8575 - recall0.3: 0.9377 - tp0.5: 7590439.7557 - fp0.5: 709361.4364 - tn0.5: 43400490.0500 - fn0.5: 810357.0554 - precision0.5: 0.9153 - recall0.5: 0.9035 - tp
    0.7: 7196681.5800 - fp0.7: 343281.3755 - tn0.7: 43766582.6877 - fn0.7: 1204115.2311 - precision0.7: 0.9549 - recall0.7: 0.8569 - tp0.9: 6477320.2436 - fp0.9: 94404.4832 - tn0.9: 44015438.6300 - fn0.9: 1923476.5675 - precision0.9: 0.985
    7 - recall0.9: 0.7709 - accuracy: 0.9710 - auc: 0.9804 - val_loss: 0.7317 - val_tp0.1: 3916023.0000 - val_fp0.1: 3484452.0000 - val_tn0.1: 18565136.0000 - val_fn0.1: 248781.0000 - val_precision0.1: 0.5292 - val_recall0.1: 0.9403 - val_
    tp0.3: 3708159.0000 - val_fp0.3: 2970626.0000 - val_tn0.3: 19078970.0000 - val_fn0.3: 456645.0000 - val_precision0.3: 0.5552 - val_recall0.3: 0.8904 - val_tp0.5: 3475102.0000 - val_fp0.5: 2669322.0000 - val_tn0.5: 19380278.0000 - val_f
    n0.5: 689702.0000 - val_precision0.5: 0.5656 - val_recall0.5: 0.8344 - val_tp0.7: 3188450.0000 - val_fp0.7: 2414665.0000 - val_tn0.7: 19634936.0000 - val_fn0.7: 976354.0000 - val_precision0.7: 0.5690 - val_recall0.7: 0.7656 - val_tp0.9
    : 2847371.0000 - val_fp0.9: 2085317.0000 - val_tn0.9: 19964284.0000 - val_fn0.9: 1317433.0000 - val_precision0.9: 0.5772 - val_recall0.9: 0.6837 - val_accuracy: 0.8719 - val_auc: 0.9139
    Epoch 42/50
    1280/1280 [==============================] - 1987s 2s/step - loss: 0.1985 - tp0.1: 8084262.2818 - fp0.1: 2648823.0039 - tn0.1: 41506455.0390 - fn0.1: 271123.8985 - precision0.1: 0.7515 - recall0.1: 0.9676 - tp0.3: 7842478.2342 - fp0.3:
     1315040.6995 - tn0.3: 42840230.2826 - fn0.3: 512907.9461 - precision0.3: 0.8555 - recall0.3: 0.9387 - tp0.5: 7555647.0101 - fp0.5: 708480.3888 - tn0.5: 43446801.7198 - fn0.5: 799739.1702 - precision0.5: 0.9137 - recall0.5: 0.9044 - tp
    0.7: 7164901.3419 - fp0.7: 337266.9789 - tn0.7: 43818009.3770 - fn0.7: 1190484.8384 - precision0.7: 0.9549 - recall0.7: 0.8575 - tp0.9: 6464041.4738 - fp0.9: 94869.2607 - tn0.9: 44060414.4278 - fn0.9: 1891344.7065 - precision0.9: 0.985
    5 - recall0.9: 0.7739 - accuracy: 0.9716 - auc: 0.9811 - val_loss: 0.5958 - val_tp0.1: 3933554.0000 - val_fp0.1: 2955615.0000 - val_tn0.1: 18990420.0000 - val_fn0.1: 334809.0000 - val_precision0.1: 0.5710 - val_recall0.1: 0.9216 - val_
    tp0.3: 3653243.0000 - val_fp0.3: 2331195.0000 - val_tn0.3: 19614844.0000 - val_fn0.3: 615120.0000 - val_precision0.3: 0.6105 - val_recall0.3: 0.8559 - val_tp0.5: 3388864.0000 - val_fp0.5: 1939453.0000 - val_tn0.5: 20006580.0000 - val_f
    n0.5: 879499.0000 - val_precision0.5: 0.6360 - val_recall0.5: 0.7939 - val_tp0.7: 3079796.0000 - val_fp0.7: 1608320.0000 - val_tn0.7: 20337718.0000 - val_fn0.7: 1188567.0000 - val_precision0.7: 0.6569 - val_recall0.7: 0.7215 - val_tp0.
    9: 2715008.0000 - val_fp0.9: 1171919.0000 - val_tn0.9: 20774120.0000 - val_fn0.9: 1553355.0000 - val_precision0.9: 0.6985 - val_recall0.9: 0.6361 - val_accuracy: 0.8925 - val_auc: 0.9240
    Epoch 43/50
    1280/1280 [==============================] - 1955s 2s/step - loss: 0.1987 - tp0.1: 8221268.6706 - fp0.1: 2670843.3326 - tn0.1: 41354847.4762 - fn0.1: 263685.1889 - precision0.1: 0.7553 - recall0.1: 0.9689 - tp0.3: 7974364.6284 - fp0.3:
     1305978.7268 - tn0.3: 42719745.8759 - fn0.3: 510589.2311 - precision0.3: 0.8607 - recall0.3: 0.9397 - tp0.5: 7690894.8962 - fp0.5: 703492.0546 - tn0.5: 43322229.0648 - fn0.5: 794058.9633 - precision0.5: 0.9176 - recall0.5: 0.9062 - tp
    0.7: 7300427.2842 - fp0.7: 340590.7377 - tn0.7: 43685101.9196 - fn0.7: 1184526.5753 - precision0.7: 0.9563 - recall0.7: 0.8599 - tp0.9: 6589497.3521 - fp0.9: 94983.5316 - tn0.9: 43930710.3302 - fn0.9: 1895456.5074 - precision0.9: 0.986
    2 - recall0.9: 0.7761 - accuracy: 0.9717 - auc: 0.9819 - val_loss: 0.6235 - val_tp0.1: 3909308.0000 - val_fp0.1: 3193225.0000 - val_tn0.1: 18785172.0000 - val_fn0.1: 326693.0000 - val_precision0.1: 0.5504 - val_recall0.1: 0.9229 - val_
    tp0.3: 3626074.0000 - val_fp0.3: 2618929.0000 - val_tn0.3: 19359480.0000 - val_fn0.3: 609927.0000 - val_precision0.3: 0.5806 - val_recall0.3: 0.8560 - val_tp0.5: 3338230.0000 - val_fp0.5: 2236986.0000 - val_tn0.5: 19741408.0000 - val_f
    n0.5: 897771.0000 - val_precision0.5: 0.5988 - val_recall0.5: 0.7881 - val_tp0.7: 2996830.0000 - val_fp0.7: 1846715.0000 - val_tn0.7: 20131684.0000 - val_fn0.7: 1239171.0000 - val_precision0.7: 0.6187 - val_recall0.7: 0.7075 - val_tp0.
    9: 2591293.0000 - val_fp0.9: 1332148.0000 - val_tn0.9: 20646248.0000 - val_fn0.9: 1644708.0000 - val_precision0.9: 0.6605 - val_recall0.9: 0.6117 - val_accuracy: 0.8804 - val_auc: 0.9182
    Epoch 44/50
    1280/1280 [==============================] - 1965s 2s/step - loss: 0.1994 - tp0.1: 8332024.2139 - fp0.1: 2677627.7361 - tn0.1: 41232032.9399 - fn0.1: 268956.1819 - precision0.1: 0.7573 - recall0.1: 0.9690 - tp0.3: 8088072.3755 - fp0.3:
     1334673.8392 - tn0.3: 42575000.3903 - fn0.3: 512908.0203 - precision0.3: 0.8587 - recall0.3: 0.9411 - tp0.5: 7804018.2927 - fp0.5: 727991.3505 - tn0.5: 43181681.2311 - fn0.5: 796962.1030 - precision0.5: 0.9148 - recall0.5: 0.9085 - tp
    0.7: 7411774.5628 - fp0.7: 351151.7166 - tn0.7: 43558520.9914 - fn0.7: 1189205.8329 - precision0.7: 0.9548 - recall0.7: 0.8632 - tp0.9: 6689648.7783 - fp0.9: 96689.1569 - tn0.9: 43812989.9953 - fn0.9: 1911331.6175 - precision0.9: 0.985
    8 - recall0.9: 0.7794 - accuracy: 0.9712 - auc: 0.9819 - val_loss: 0.5577 - val_tp0.1: 3767211.0000 - val_fp0.1: 2896593.0000 - val_tn0.1: 19207012.0000 - val_fn0.1: 343594.0000 - val_precision0.1: 0.5653 - val_recall0.1: 0.9164 - val_
    tp0.3: 3483824.0000 - val_fp0.3: 2230299.0000 - val_tn0.3: 19873292.0000 - val_fn0.3: 626981.0000 - val_precision0.3: 0.6097 - val_recall0.3: 0.8475 - val_tp0.5: 3200515.0000 - val_fp0.5: 1719966.0000 - val_tn0.5: 20383632.0000 - val_f
    n0.5: 910290.0000 - val_precision0.5: 0.6504 - val_recall0.5: 0.7786 - val_tp0.7: 2881571.0000 - val_fp0.7: 1322038.0000 - val_tn0.7: 20781552.0000 - val_fn0.7: 1229234.0000 - val_precision0.7: 0.6855 - val_recall0.7: 0.7010 - val_tp0.
    9: 2488169.0000 - val_fp0.9: 864656.0000 - val_tn0.9: 21238940.0000 - val_fn0.9: 1622636.0000 - val_precision0.9: 0.7421 - val_recall0.9: 0.6053 - val_accuracy: 0.8997 - val_auc: 0.9249
    Epoch 45/50
    1280/1280 [==============================] - 1967s 2s/step - loss: 0.2054 - tp0.1: 8039884.9914 - fp0.1: 2657730.7822 - tn0.1: 41537257.0453 - fn0.1: 275776.4333 - precision0.1: 0.7479 - recall0.1: 0.9656 - tp0.3: 7799006.9430 - fp0.3:
     1331880.2607 - tn0.3: 42863099.0406 - fn0.3: 516654.4817 - precision0.3: 0.8527 - recall0.3: 0.9360 - tp0.5: 7508282.6300 - fp0.5: 716584.7447 - tn0.5: 43478405.7619 - fn0.5: 807378.7947 - precision0.5: 0.9120 - recall0.5: 0.9006 - tp
    0.7: 7111748.2311 - fp0.7: 345919.2553 - tn0.7: 43849114.1967 - fn0.7: 1203913.1936 - precision0.7: 0.9530 - recall0.7: 0.8525 - tp0.9: 6406401.2319 - fp0.9: 97963.5379 - tn0.9: 44097016.4520 - fn0.9: 1909260.1928 - precision0.9: 0.984
    7 - recall0.9: 0.7671 - accuracy: 0.9708 - auc: 0.9799 - val_loss: 0.5755 - val_tp0.1: 3947505.0000 - val_fp0.1: 3387487.0000 - val_tn0.1: 18700156.0000 - val_fn0.1: 179254.0000 - val_precision0.1: 0.5382 - val_recall0.1: 0.9566 - val_
    tp0.3: 3738020.0000 - val_fp0.3: 2730917.0000 - val_tn0.3: 19356726.0000 - val_fn0.3: 388739.0000 - val_precision0.3: 0.5778 - val_recall0.3: 0.9058 - val_tp0.5: 3505678.0000 - val_fp0.5: 2359050.0000 - val_tn0.5: 19728584.0000 - val_f
    n0.5: 621081.0000 - val_precision0.5: 0.5978 - val_recall0.5: 0.8495 - val_tp0.7: 3196056.0000 - val_fp0.7: 1989619.0000 - val_tn0.7: 20098016.0000 - val_fn0.7: 930703.0000 - val_precision0.7: 0.6163 - val_recall0.7: 0.7745 - val_tp0.9
    : 2807022.0000 - val_fp0.9: 1495603.0000 - val_tn0.9: 20592028.0000 - val_fn0.9: 1319737.0000 - val_precision0.9: 0.6524 - val_recall0.9: 0.6802 - val_accuracy: 0.8863 - val_auc: 0.9356
    Epoch 46/50
    1280/1280 [==============================] - 1954s 2s/step - loss: 0.2006 - tp0.1: 8213399.9953 - fp0.1: 2647494.8681 - tn0.1: 41382018.9024 - fn0.1: 267742.8033 - precision0.1: 0.7532 - recall0.1: 0.9684 - tp0.3: 7974876.2194 - fp0.3:
     1321606.3544 - tn0.3: 42707910.7440 - fn0.3: 506266.5792 - precision0.3: 0.8563 - recall0.3: 0.9401 - tp0.5: 7686530.4450 - fp0.5: 705127.5909 - tn0.5: 43324387.6230 - fn0.5: 794612.3536 - precision0.5: 0.9155 - recall0.5: 0.9060 - tp
    0.7: 7293262.6581 - fp0.7: 337606.6417 - tn0.7: 43691898.0304 - fn0.7: 1187880.1405 - precision0.7: 0.9556 - recall0.7: 0.8595 - tp0.9: 6593016.1983 - fp0.9: 93813.7900 - tn0.9: 43935696.6682 - fn0.9: 1888126.6003 - precision0.9: 0.985
    9 - recall0.9: 0.7775 - accuracy: 0.9713 - auc: 0.9815 - val_loss: 0.5779 - val_tp0.1: 3847918.0000 - val_fp0.1: 3630042.0000 - val_tn0.1: 18523056.0000 - val_fn0.1: 213382.0000 - val_precision0.1: 0.5146 - val_recall0.1: 0.9475 - val_
    tp0.3: 3624637.0000 - val_fp0.3: 2927247.0000 - val_tn0.3: 19225848.0000 - val_fn0.3: 436663.0000 - val_precision0.3: 0.5532 - val_recall0.3: 0.8925 - val_tp0.5: 3387759.0000 - val_fp0.5: 2442181.0000 - val_tn0.5: 19710918.0000 - val_f
    n0.5: 673541.0000 - val_precision0.5: 0.5811 - val_recall0.5: 0.8342 - val_tp0.7: 3093355.0000 - val_fp0.7: 1959305.0000 - val_tn0.7: 20193798.0000 - val_fn0.7: 967945.0000 - val_precision0.7: 0.6122 - val_recall0.7: 0.7617 - val_tp0.9
    : 2709194.0000 - val_fp0.9: 1315902.0000 - val_tn0.9: 20837192.0000 - val_fn0.9: 1352106.0000 - val_precision0.9: 0.6731 - val_recall0.9: 0.6671 - val_accuracy: 0.8811 - val_auc: 0.9327
    Epoch 47/50
    1280/1280 [==============================] - 1945s 2s/step - loss: 0.1969 - tp0.1: 8300923.1351 - fp0.1: 2610298.6073 - tn0.1: 41331766.4083 - fn0.1: 267669.1296 - precision0.1: 0.7642 - recall0.1: 0.9694 - tp0.3: 8061644.4340 - fp0.3:
     1313054.2631 - tn0.3: 42629006.5870 - fn0.3: 506947.8306 - precision0.3: 0.8611 - recall0.3: 0.9422 - tp0.5: 7779027.9711 - fp0.5: 711910.2787 - tn0.5: 43230141.7112 - fn0.5: 789564.2935 - precision0.5: 0.9166 - recall0.5: 0.9097 - tp
    0.7: 7390637.0796 - fp0.7: 345621.5621 - tn0.7: 43596452.1616 - fn0.7: 1177955.1850 - precision0.7: 0.9553 - recall0.7: 0.8650 - tp0.9: 6688557.9680 - fp0.9: 97298.4106 - tn0.9: 43844794.7112 - fn0.9: 1880034.2966 - precision0.9: 0.985
    5 - recall0.9: 0.7835 - accuracy: 0.9714 - auc: 0.9821 - val_loss: 0.6259 - val_tp0.1: 3930823.0000 - val_fp0.1: 3136289.0000 - val_tn0.1: 18868696.0000 - val_fn0.1: 278584.0000 - val_precision0.1: 0.5562 - val_recall0.1: 0.9338 - val_
    tp0.3: 3682497.0000 - val_fp0.3: 2597221.0000 - val_tn0.3: 19407772.0000 - val_fn0.3: 526910.0000 - val_precision0.3: 0.5864 - val_recall0.3: 0.8748 - val_tp0.5: 3429169.0000 - val_fp0.5: 2237642.0000 - val_tn0.5: 19767344.0000 - val_f
    n0.5: 780238.0000 - val_precision0.5: 0.6051 - val_recall0.5: 0.8146 - val_tp0.7: 3114765.0000 - val_fp0.7: 1889104.0000 - val_tn0.7: 20115888.0000 - val_fn0.7: 1094642.0000 - val_precision0.7: 0.6225 - val_recall0.7: 0.7400 - val_tp0.
    9: 2710793.0000 - val_fp0.9: 1416123.0000 - val_tn0.9: 20588868.0000 - val_fn0.9: 1498614.0000 - val_precision0.9: 0.6569 - val_recall0.9: 0.6440 - val_accuracy: 0.8849 - val_auc: 0.9224
    Epoch 48/50
    1280/1280 [==============================] - 1953s 2s/step - loss: 0.1928 - tp0.1: 8105003.8283 - fp0.1: 2580803.4801 - tn0.1: 41568825.8158 - fn0.1: 256048.8673 - precision0.1: 0.7573 - recall0.1: 0.9697 - tp0.3: 7868496.5355 - fp0.3:
     1297144.6347 - tn0.3: 42852437.2209 - fn0.3: 492556.1600 - precision0.3: 0.8575 - recall0.3: 0.9414 - tp0.5: 7589914.3154 - fp0.5: 704118.4832 - tn0.5: 43445489.9368 - fn0.5: 771138.3802 - precision0.5: 0.9146 - recall0.5: 0.9080 - tp
    0.7: 7204444.4348 - fp0.7: 338098.1210 - tn0.7: 43811497.8689 - fn0.7: 1156608.2607 - precision0.7: 0.9549 - recall0.7: 0.8617 - tp0.9: 6516177.3638 - fp0.9: 93695.1686 - tn0.9: 44055912.6136 - fn0.9: 1844875.3318 - precision0.9: 0.985
    8 - recall0.9: 0.7794 - accuracy: 0.9719 - auc: 0.9822 - val_loss: 0.5549 - val_tp0.1: 3956721.0000 - val_fp0.1: 3232246.0000 - val_tn0.1: 18771428.0000 - val_fn0.1: 254013.0000 - val_precision0.1: 0.5504 - val_recall0.1: 0.9397 - val_
    tp0.3: 3708830.0000 - val_fp0.3: 2603884.0000 - val_tn0.3: 19399780.0000 - val_fn0.3: 501904.0000 - val_precision0.3: 0.5875 - val_recall0.3: 0.8808 - val_tp0.5: 3455265.0000 - val_fp0.5: 2129156.0000 - val_tn0.5: 19874504.0000 - val_f
    n0.5: 755469.0000 - val_precision0.5: 0.6187 - val_recall0.5: 0.8206 - val_tp0.7: 3150570.0000 - val_fp0.7: 1691047.0000 - val_tn0.7: 20312616.0000 - val_fn0.7: 1060164.0000 - val_precision0.7: 0.6507 - val_recall0.7: 0.7482 - val_tp0.
    9: 2734733.0000 - val_fp0.9: 1128005.0000 - val_tn0.9: 20875662.0000 - val_fn0.9: 1476001.0000 - val_precision0.9: 0.7080 - val_recall0.9: 0.6495 - val_accuracy: 0.8900 - val_auc: 0.9326
    Epoch 49/50
    1280/1280 [==============================] - 1952s 2s/step - loss: 0.1904 - tp0.1: 8205918.2201 - fp0.1: 2597375.0523 - tn0.1: 41463883.8283 - fn0.1: 243494.6815 - precision0.1: 0.7596 - recall0.1: 0.9713 - tp0.3: 7967288.4333 - fp0.3:
     1294707.1905 - tn0.3: 42766530.0679 - fn0.3: 482124.4684 - precision0.3: 0.8602 - recall0.3: 0.9426 - tp0.5: 7690726.4028 - fp0.5: 706852.2178 - tn0.5: 43354411.7049 - fn0.5: 758686.4988 - precision0.5: 0.9160 - recall0.5: 0.9099 - tp
    0.7: 7310265.9742 - fp0.7: 341418.8126 - tn0.7: 43719824.0367 - fn0.7: 1139146.9274 - precision0.7: 0.9557 - recall0.7: 0.8651 - tp0.9: 6619945.5667 - fp0.9: 95254.3942 - tn0.9: 43965990.5909 - fn0.9: 1829467.3349 - precision0.9: 0.986
    3 - recall0.9: 0.7836 - accuracy: 0.9719 - auc: 0.9831 - val_loss: 0.7113 - val_tp0.1: 3891189.0000 - val_fp0.1: 3192765.0000 - val_tn0.1: 18774752.0000 - val_fn0.1: 355695.0000 - val_precision0.1: 0.5493 - val_recall0.1: 0.9162 - val_
    tp0.3: 3627202.0000 - val_fp0.3: 2690459.0000 - val_tn0.3: 19277064.0000 - val_fn0.3: 619682.0000 - val_precision0.3: 0.5741 - val_recall0.3: 0.8541 - val_tp0.5: 3369672.0000 - val_fp0.5: 2384023.0000 - val_tn0.5: 19583492.0000 - val_f
    n0.5: 877212.0000 - val_precision0.5: 0.5857 - val_recall0.5: 0.7934 - val_tp0.7: 3076485.0000 - val_fp0.7: 2107024.0000 - val_tn0.7: 19860492.0000 - val_fn0.7: 1170399.0000 - val_precision0.7: 0.5935 - val_recall0.7: 0.7244 - val_tp0.
    9: 2710033.0000 - val_fp0.9: 1728311.0000 - val_tn0.9: 20239206.0000 - val_fn0.9: 1536851.0000 - val_precision0.9: 0.6106 - val_recall0.9: 0.6381 - val_accuracy: 0.8756 - val_auc: 0.9085
    Epoch 50/50
    1280/1280 [==============================] - 1959s 2s/step - loss: 0.1904 - tp0.1: 8168142.8899 - fp0.1: 2531078.8610 - tn0.1: 41557584.1272 - fn0.1: 253833.5800 - precision0.1: 0.7637 - recall0.1: 0.9699 - tp0.3: 7935448.9009 - fp0.3:
     1268230.0484 - tn0.3: 42820446.9766 - fn0.3: 486527.5691 - precision0.3: 0.8629 - recall0.3: 0.9420 - tp0.5: 7667088.7853 - fp0.5: 699134.4457 - tn0.5: 43389534.5613 - fn0.5: 754887.6846 - precision0.5: 0.9167 - recall0.5: 0.9102 - tp
    0.7: 7290487.3372 - fp0.7: 342110.9336 - tn0.7: 43746573.7869 - fn0.7: 1131489.1327 - precision0.7: 0.9553 - recall0.7: 0.8655 - tp0.9: 6585906.2607 - fp0.9: 97053.1694 - tn0.9: 43991631.9688 - fn0.9: 1836070.2092 - precision0.9: 0.985
    5 - recall0.9: 0.7818 - accuracy: 0.9724 - auc: 0.9826 - val_loss: 0.5552 - val_tp0.1: 3972788.0000 - val_fp0.1: 3022431.0000 - val_tn0.1: 18950448.0000 - val_fn0.1: 268737.0000 - val_precision0.1: 0.5679 - val_recall0.1: 0.9366 - val_
    tp0.3: 3744205.0000 - val_fp0.3: 2467134.0000 - val_tn0.3: 19505742.0000 - val_fn0.3: 497320.0000 - val_precision0.3: 0.6028 - val_recall0.3: 0.8827 - val_tp0.5: 3515241.0000 - val_fp0.5: 2098091.0000 - val_tn0.5: 19874784.0000 - val_f
    n0.5: 726284.0000 - val_precision0.5: 0.6262 - val_recall0.5: 0.8288 - val_tp0.7: 3227939.0000 - val_fp0.7: 1707258.0000 - val_tn0.7: 20265616.0000 - val_fn0.7: 1013586.0000 - val_precision0.7: 0.6541 - val_recall0.7: 0.7610 - val_tp0.
    9: 2849341.0000 - val_fp0.9: 1228067.0000 - val_tn0.9: 20744808.0000 - val_fn0.9: 1392184.0000 - val_precision0.9: 0.6988 - val_recall0.9: 0.6718 - val_accuracy: 0.8923 - val_auc: 0.9339
    420/420 [==============================] - 126s 300ms/step - loss: 0.5623 - tp0.1: 4784751.0000 - fp0.1: 3873965.0000 - tn0.1: 25422992.0000 - fn0.1: 324680.0000 - precision0.1: 0.5526 - recall0.1: 0.9365 - tp0.3: 4514069.0000 - fp0.3:
     3235942.0000 - tn0.3: 26061030.0000 - fn0.3: 595362.0000 - precision0.3: 0.5825 - recall0.3: 0.8835 - tp0.5: 4242444.0000 - fp0.5: 2843465.0000 - tn0.5: 26453508.0000 - fn0.5: 866987.0000 - precision0.5: 0.5987 - recall0.5: 0.8303 - t
    p0.7: 3892738.0000 - fp0.7: 2409415.0000 - tn0.7: 26887552.0000 - fn0.7: 1216693.0000 - precision0.7: 0.6177 - recall0.7: 0.7619 - tp0.9: 3413153.0000 - fp0.9: 1853895.0000 - tn0.9: 27443078.0000 - fn0.9: 1696278.0000 - precision0.9: 0
    .6480 - recall0.9: 0.6680 - accuracy: 0.8922 - auc: 0.9306
    2021/02/06 23:57:14 INFO mlflow.projects: === Run (ID 'b9935d1e554c423fb2852242f4c4504c') succeeded ===
    (tf-nightly) [ye53nis@node221 drmed-git]$
    

2.3.3 Read out logs of Run 1

2.3.3.1 test dataset statistics
  • test data is not saved out automatically, but can be copied from the log above

    420/420 [==============================] - 126s 300ms/step - loss: 0.5623

    0.1 0.3 0.5 0.7 0.9
    tp 4784751.0000 4514069.0000 4242444.0000 3892738.0000 3413153.0000
    fp 3873965.0000 3235942.0000 2843465.0000 2409415.0000 1853895.0000
    fn 324680.0000 595362.0000 866987.0000 1216693.0000 1696278.0000
    tn 25422992.0000 26061030.0000 26453508.0000 26887552.0000 27443078.0000
    all 34,406,388 34,406,403 34,406,404 34,406,398 34,406,404
    precision 0.5526 0.5825 0.5987 0.6177 0.6480
    recall 0.9365 0.8835 0.8303 0.7619 0.6680

    accuracy: 0.8922 - auc: 0.9306

    0.1 actual positive actual negative  
    pred positive 0.13906577 0.11259435 Prec: 0.5526
    pred negative 9.4345271e-3 0.73890325  
      Recall: 0.9365    
          F1: 0.69506400
    0.3 actual positive actual negative  
    pred positive 0.13119852 0.094050575 Prec: 0.5825
    pred negative 0.017303814 0.75744709  
      Recall: 0.8835    
          F1: 0.70209925
    0.5 actual positive actual negative  
    pred positive 0.12330391 0.082643481 Prec: 0.5987
    pred negative 0.025198419 0.76885419  
      Recall: 0.8303    
          F1: 0.69573213
    0.7 actual positive actual negative  
    pred positive 0.11313995 0.070028109 Prec: 0.6177
    pred negative 0.035362406 0.78146954  
      Recall: 0.7619    
          F1: 0.68226389
    0.9 actual positive actual negative  
    pred positive 0.099201096 0.053882266 Prec: 0.6480
    pred negative 0.049301229 0.79761541  
      Recall: 0.6680    
          F1: 0.65784802
2.3.3.2 prediction plots after each epoch

after epoch 1: plot0.png epoch 2: plot1.png epoch 3: plot2.png epoch 4: plot3.png epoch 5: plot4.png epoch 10: plot9.png epoch 20: plot19.png epoch 30: plot29.png epoch 40: plot39.png epoch 50: plot49.png

2.3.3.3 Git log after some code additions
!git log -10
commit a070d3b531725e0fb37688dde80e990083ccf1cc
Author: Apoplex <oligolex@vivaldi.net>
Date:   Mon Mar 15 21:48:01 2021 +0100

    Fix photon count bin metadata 2

commit 86bcbfe13f4026f9b554ba8f0c7e3b9360090331
Author: Apoplex <oligolex@vivaldi.net>
Date:   Mon Mar 15 21:40:47 2021 +0100

    Fix photon count bin metadata
commit 1c7ca995b2e550f6b6cb71dca647f170bbd9982d
Author: Apoplex <oligolex@vivaldi.net>
Date:   Mon Mar 15 19:10:01 2021 +0100

    Fix correction by list of prediction thresholds

commit 61cbc69dc8be177a754afab9dfea82ab6cd1086e
Author: Apoplex <oligolex@vivaldi.net>
Date:   Mon Mar 15 19:03:07 2021 +0100

    Add correction by different prediction thresholds

commit 58aa20f75747c43056ebe56831395b01c0535842
Author: Apoplex <oligolex@vivaldi.net>
Date:   Mon Mar 15 11:04:31 2021 +0100

    Fix column name \tau

commit 01c14384c452ac4106263d088e0587ddf4ebc379
Author: Apoplex <oligolex@vivaldi.net>
Date:   Mon Mar 8 18:33:48 2021 +0100

    Add FileNotFoundError checks to import functions

commit 28640d1360ca77aaa633de04e3349da6f12b208c
Author: Apoplex <oligolex@vivaldi.net>
Date:   Mon Mar 8 17:45:33 2021 +0100

    Fix if ntraces=None

commit 5e493d6c826ca24b425a3b11c0b07d31aac8af1a
Author: Apoplex <oligolex@vivaldi.net>
Date:   Mon Mar 8 17:17:42 2021 +0100

    Add correction function for ptu files + csv export

commit 61f076ecec76dd1f7c0323feb2b9e48f60368582
Author: Apoplex <oligolex@vivaldi.net>
Date:   Sat Feb 20 23:43:15 2021 +0100

    Fix concatenation order, add missing traces note

commit ed974cb634e7e1c47b730b136fbd09d9ee20535f
Author: Apoplex <oligolex@vivaldi.net>
Date:   Sat Feb 20 22:35:14 2021 +0100

    Fix np.repeat, remove nfiles, print no of traces
2.3.3.4 Application 1 - load modules, set parameters
  1. Load the required modules
    %cd /beegfs/ye53nis/drmed-git
    
    /beegfs/ye53nis/drmed-git
    
    from pathlib import Path
    
    import sys
    import mlflow
    import matplotlib.pyplot as plt
    import numpy as np
    import pandas as pd
    import seaborn as sns
    mlflow.version.VERSION
    
    1.13.1
    
    sys.path.append('src/')
    from fluotracify.simulations import (
       import_simulation_from_csv as isfc,
       analyze_simulations as ans,
    )
    from fluotracify.training import build_model as bm, preprocess_data as ppd
    from fluotracify.applications import correlate, plots, correction
    from fluotracify.imports import ptu_utils as ptu
    
    import importlib
    importlib.reload(correction)
    
    <module 'fluotracify.applications.correction' from 'src/fluotracify/applications/correction.py'>
    
folder = '/beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/'
col_per_example = 3
lab_thresh = 0.04
pred_thresh = 0.5
xunit = 1
artifact = 0
model_type = 1
fwhm = 250
run_id = 'b9935d1e554c423fb2852242f4c4504c'
length_delimiter = 2**14
  • now load the trained model
    mlflow.set_tracking_uri('file:///beegfs/ye53nis/drmed-git/data/mlruns')
    client = mlflow.tracking.MlflowClient(tracking_uri=mlflow.get_tracking_uri())
    model_path = client.download_artifacts(run_id=run_id,
                                           path='model')
    model_keras = mlflow.keras.load_model(model_uri=model_path,
                                          custom_objects={'binary_ce_dice':bm.binary_ce_dice_loss()})
    print(model_path, '\n', model_keras)
    
    /beegfs/ye53nis/drmed-git/data/mlruns/3/b9935d1e554c423fb2852242f4c4504c/artifacts/model
     <tensorflow.python.keras.engine.functional.Functional object at 0x2ae3e21c2d60>
    
2.3.3.5 Application 2 - test data
  • I copied the test data which was randomly sampled in the mlflow run (See docs above) in an extra directory. Next time, train and test data will be split beforehand.
    dataset, _, nsamples, experiment_params = isfc.import_from_csv(
        folder=folder,
        header=12,
        frac_train=1,
        col_per_example=col_per_example,
        dropindex=None,
        dropcolumns=None)
    experiment_params
    
    train 0 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D3.0_set009.csv
    train 1 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.08_set003.csv
    train 2 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D50_set003.csv
    train 3 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.4_set007.csv
    train 4 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.6_set009.csv
    train 5 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D3.0_set008.csv
    train 6 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.2_set010.csv
    train 7 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D1.0_set008.csv
    train 8 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.2_set007.csv
    train 9 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.1_set002.csvtrain 10 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.2_set008.csv
    train 11 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D10_set001.csv
    train 12 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.4_set008.csv
    train 13 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.4_set005.csv
    train 14 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D3.0_set004.csv
    train 15 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D1.0_set005.csv
    train 16 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.1_set006.csv
    train 17 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D0.069_set010.csv
    train 18 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D3.0_set002.csv
    train 19 /beegfs/ye53nis/saves/firstartifact_Nov2020_test_run1/traces_brightclust_Nov2020_D1.0_set006.csv
    
      0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
    unique identifier 340ad302-cf12-4224-addf-9455009bc9b3 b85aeec3-5e79-444f-b0c5-5ba7057fad8c b2b78a0f-278b-4b7d-82a3-5e625fbc91de 32d6e80c-92af-48c2-a756-1eee94d17c74 87d9374a-3ae4-4dfb-8dc3-ec401df4961d 90ec2bbf-d308-4895-b5fd-1974c0b3390d 5c310958-a0cc-4d1d-b721-f8652afd9707 b5d1537c-4113-4f13-a1ca-65664181f8cc 33ee5002-7064-4882-88de-79c773d741cd 9a2f89bd-c64e-4324-910c-46a5b8a657ba 71894018-b950-4672-a182-653d59f3b7db 6a3c00e3-24e6-4758-9c6d-f2a9dc23f7d9 20f0a9df-2c4d-409f-86de-ceda40d1aafe eb3f3035-955f-4e86-bfbd-16421eede63f cf57c545-933f-41b8-bcfe-19723d79b7bf 84116b56-7264-4f63-833a-2fa1af968288 55e826c0-a09a-4f1e-8f57-ddc113715aaf bebf4d6b-0ffe-4dcf-be78-93bfa70dd04c f2dcf7ee-f180-4b26-9706-526ceb6b9fe6 c1a85eb4-f202-40f7-8bce-cd3fd3a843dc
    path and file name /beegfs/ye53nis/saves/firstartifact\Nov2020/3…. /beegfs/ye53nis/saves/firstartifact\Nov2020/0…. /beegfs/ye53nis/saves/firstartifact\Nov2020/50… /beegfs/ye53nis/saves/firstartifact\Nov2020/0…. /beegfs/ye53nis/saves/firstartifact\Nov2020/0…. /beegfs/ye53nis/saves/firstartifact\Nov2020/3…. /beegfs/ye53nis/saves/firstartifact\Nov2020/0…. /beegfs/ye53nis/saves/firstartifact\Nov2020/1…. /beegfs/ye53nis/saves/firstartifact\Nov2020/0…. /beegfs/ye53nis/saves/firstartifact\Nov2020/0…. /beegfs/ye53nis/saves/firstartifact\Nov2020/0…. /beegfs/ye53nis/saves/firstartifact\Nov2020/10… /beegfs/ye53nis/saves/firstartifact\Nov2020/0…. /beegfs/ye53nis/saves/firstartifact\Nov2020/0…. /beegfs/ye53nis/saves/firstartifact\Nov2020/3…. /beegfs/ye53nis/saves/firstartifact\Nov2020/1…. /beegfs/ye53nis/saves/firstartifact\Nov2020/0…. /beegfs/ye53nis/saves/firstartifact\Nov2020/0…. /beegfs/ye53nis/saves/firstartifact\Nov2020/3…. /beegfs/ye53nis/saves/firstartifact\Nov2020/1….
    FWHMs of excitation PSFs used in nm [250] [250] [250] [250] [250] [250] [250] [250] [250] [250] [250] [250] [250] [250] [250] [250] [250] [250] [250] [250]
    Extent of simulated PSF (distance to center of Gaussian) in nm 4000 4000 4000 4000 4000 4000 4000 4000 4000 4000 4000 4000 4000 4000 4000 4000 4000 4000 4000 4000
    total simulation time in ms 16384 16384 16384 16384 16384 16384 16384 16384 16384 16384 16384 16384 16384 16384 16384 16384 16384 16384 16384 16384
    time step in ms 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0
    number of fast molecules 2422 856 2561 2887 1296 766 1247 3300 895 2975 1239 1376 1724 560 1785 2059 2060 679 2900 1357
    diffusion rate of molecules in micrometer\2 / s 3.0 0.08 50 0.4 0.6 3.0 0.2 1.0 0.2 0.1 0.2 10 0.4 0.4 3.0 1.0 0.1 0.069 3.0 1.0
    width of the simulation in nm 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0
    height of the simulation in nm 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0 3000.0
    number of slow clusters 3 7 7 3 3 10 3 3 7 10 7 7 10 3 10 3 10 3 3 10
    diffusion rate of clusters in micrometer\2 / s 1.0 0.1 0.1 1.0 1.0 0.01 1.0 1.0 0.1 0.01 0.1 0.1 0.01 1.0 0.01 1.0 0.01 1.0 1.0 0.01
    trace001 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1 label001\1
    diffrates = experiment_params.loc['diffusion rate of molecules in micrometer^2 / s'].astype(np.float32)
    nmols = experiment_params.loc['number of fast molecules'].astype(np.float32)
    clusters = experiment_params.loc['diffusion rate of clusters in micrometer^2 / s'].astype(np.float32)
    
    dataset_sep = isfc.separate_data_and_labels(array=dataset,
                                                nsamples=nsamples,
                                                col_per_example=col_per_example)
    
    features = dataset_sep['0']
    labels_artifact = dataset_sep['1']
    labels_artifact_bool = labels_artifact > lab_thresh
    labels_puretrace = dataset_sep['2']
    
    The given DataFrame was split into 3 parts with shapes: [(16384, 2000), (16384, 2000), (16384, 2000)]
    
  • First attempt: look at distribution of all correlated diffrates by each simulated diffusion rate. The following function was thus executed for each simulated diffusion time and the box plots put together in inkscape by hand.
    pred_thresh = 0.5
    diffrate_of_interest = 50
    transit_time_expected = ((float(fwhm) / 1000)**2 * 1000) / (
        diffrate_of_interest * 8 * np.log(2.0))
    upper_bin = np.ceil(4*transit_time_expected)
    xunit_bins = np.arange(0, upper_bin, upper_bin / 50)
    
    out = plots.plot_distribution_of_correlations_corrected_by_prediction(
        diffrate_of_interest=diffrate_of_interest,
        model=model_keras,
        lab_thresh=lab_thresh,
        pred_thresh=pred_thresh,
        xunit=xunit,
        artifact=artifact,
        model_type=model_type,
        xunit_bins=xunit_bins,
        experiment_params=experiment_params,
        nsamples=nsamples,
        features=features,
        labels_artifact=labels_artifact,
        labels_puretrace=labels_puretrace,
        number_of_traces=100)
    

    210215_artifact0_xunit1_alldiffrates.svg

  • We see some interesting results in the corrupted data without correction (first box, in blue), can they be explained by the test data?
    speed of clusters 0.01 0.1 1.0
    0.069 test run1     10
    0.08 test run1   3  
    0.1 test run1 2, 6    
    0.2 test run1   7, 8 10
    0.4 test run1 8   5, 7
    0.6 test run1     9
    1.0 test run1 6   5, 8
    3.0 test run1 4, 8   2, 9
    10 test run1   1  
    50 test run1   3  
    • subsets which look like the bright clusters lead to a shift in the correlation curves towards their own transit time:
      • 0.069=163.35ms: cluster speed 1.0=11.27ms, median correlated speed 11.6ms
      • 0.08=140.89ms: cluster speed 0.1=112.71ms, median correlated speed 100.5ms
      • 0.6=18.79ms: cluster speed 1.0=11.27ms, median correlated speed 12.10ms
      • 10=1.13ms: cluster speed 0.1=112.71ms, median correlated speed 92.1ms
      • 50=0.23ms: cluster speed 0.1=112.71ms, median correlated speed 96.5ms
    • subsets which fall out of the pattern where I have to take a closer look:
      • 0.1=112.71ms: cluster speed 0.01=1127ms, median correlated speed 346.6ms → possible explanation: it is a rule of thumb in the FCS community, that you need around 2-3 orders of magnitude of measurement time longer than transit times you want to measure. For speeds of 1.13s that means measurement times of 113-1130s, where we are far away from with our simulated ~16s of trace.
      • 0.2=56.36ms: cluster speed 0.1=112.71ms AND 1.0=11.27ms, median correlated speed 73ms → possible explanation: there seem to be two maxima in the distribution, one around 10ms, one around 70-110ms, which kind of fits to the two kinds of clusters in these traces. → another strange thing: it has a quite long tail of transit times, way over 200ms, which does not seem related to the direct cluster speeds.
      • 0.4=28.18ms, 1.0=11.27ms and 3.0=3.76ms: cluster speed 0.01=1127ms AND 1.0=11.27ms, median correlated speed 14.4ms → strange case, since the traces with the slower 0.01 clusters do not show up in the distribution! The log of isfc.import_from_csv shows, that the respective files were loaded and used for these plots.
  • Looking at the predictions (and controls)
    • subsets which look fine, as in: predictions ~ controls
      • 0.069, 0.08, 0.2, 0.6
    • subsets where there seem to be a subsubset of traces where the prediction fails:
      • 0.1: median is fine, but the distribution is a lot wider, including a considerable left-ward skew, which does not show in either of the other distributions (corrupted or controls)
      • 0.4: median is fine, distribution is also okay, but a subset of traces gets very low transit times after correlation → have to figure out which subset this is
      • 1.0, 3.0: median is fine, but distribution has a considerable left-ward skew, and quite some outliers on the right
      • 10, 50: median and distribution are okay, BUT here there is quite a difference between the distribution in predictions, label control and pure control → here, most clearly in 50, it seems artifacts are introduced through the correction method.
corr_out = ans.correlate_simulations_corrected_by_prediction(
    model=model_keras,
    lab_thresh=lab_thresh,
    pred_thresh=pred_thresh,
    artifact=artifact,
    model_type=model_type,
    experiment_params=experiment_params,
    nsamples=nsamples,
    features=features,
    labels_artifact=labels_artifact,
    labels_puretrace=labels_puretrace,
    save_as_csv=True)
corr_out
processed correlation of 2000 traces with correction by label
processed correlation of 1999 traces with correction by prediction
processed correlation of 2000 traces without correction
processed correlation of pure 2000 traces
  Simulated \(D\) Simulated \(D\_{{clust}}\) nmol \(D\) in \(\frac{{\mu m\^2}}{{s}}\) \(\tau\_{{D}}\) in \(ms\) Trace lengths Traces used
0 3.0 1.0 2422.0 1.260146 8.944242 16384 corrupted without correction
1 3.0 1.0 2422.0 1.156435 9.746384 16384 corrupted without correction
2 3.0 1.0 2422.0 1.716206 6.567426 16384 corrupted without correction
3 3.0 1.0 2422.0 0.999673 11.274746 16384 corrupted without correction
4 3.0 1.0 2422.0 1.040372 10.833679 16384 corrupted without correction
7995 1.0 0.01 1357.0 0.706696 15.94895 14933 corrected by prediction
7996 1.0 0.01 1357.0 0.765425 14.725227 11550 corrected by prediction
7997 1.0 0.01 1357.0 2.033196 5.543517 1250 corrected by prediction
7998 1.0 0.01 1357.0 3.744989 3.009636 261 corrected by prediction
7999 1.0 0.01 1357.0 None None None corrected by prediction

8000 rows × 7 columns

corr_out = pd.read_csv(filepath_or_buffer='data/exp-210204-unet/2021-02-20_correlations.csv')
corr_out
  Simulated \(D\) Simulated \(D\_{{clust}}\) nmol \(D\) in \(\frac{{\mu m\^2}}{{s}}\) \(\tau\_{{D}}\) in \(ms\) Trace lengths Traces used
0 3.0 1.00 2422.0 1.260146 8.944242 16384.0 corrupted without correction
1 3.0 1.00 2422.0 1.156435 9.746384 16384.0 corrupted without correction
2 3.0 1.00 2422.0 1.716206 6.567426 16384.0 corrupted without correction
3 3.0 1.00 2422.0 0.999673 11.274746 16384.0 corrupted without correction
4 3.0 1.00 2422.0 1.040372 10.833679 16384.0 corrupted without correction
7995 1.0 0.01 1357.0 0.706696 15.948950 14933.0 corrected by prediction
7996 1.0 0.01 1357.0 0.765425 14.725227 11550.0 corrected by prediction
7997 1.0 0.01 1357.0 2.033196 5.543517 1250.0 corrected by prediction
7998 1.0 0.01 1357.0 3.744989 3.009636 261.0 corrected by prediction
7999 1.0 0.01 1357.0 NaN NaN NaN corrected by prediction

8000 rows × 7 columns

Awesome! Now let's try categorical plotting to examine the results in a more structured manner. For the following 3 plots, I just changed the x value.

ax = sns.catplot(data=corr_out,
      kind='strip',
      x='$D$ in $\\frac{{\mu m^2}}{{s}}$',
      y='Traces used',
      hue='Simulated $D_{{clust}}$',
      col='Simulated $D$',
      col_wrap=3,
      orient='h',
      dodge=True,
      sharex=False)
plt.show()

Trace lengths: 2021-02-20_correlations_trace-lengths.png Transit times: 2021-02-20_correlations_transittimes.png Diffusion rates: 2021-02-20_correlations_diffusionrates.png

I have the theory, that trace lengths have quite an impact on transit times and diffusion rates. Let's try to do a big plot of transittimes vs trace lengths and diffusion rates against trace lengths.

2.3.3.6 Application 3 - experimental data

Here I used the code in a jupyter notebook to produce this plot: ptu_brightbursts_correction_by_unet_histogram_010_tt_210303_400traces.svg

  1. There is a problem with the UNet, if it has to classify features different than the training size (1, 16384, 1) for the first time. It gives the following error:
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-7-4c721240eb1b> in <module>
          3 length_delimiter = 2**13  # for U-Net
          4 bin_for_correlation = 1e5
    ----> 5 out = correction.correct_experimental_traces_from_ptu_by_unet_prediction(
          6   path_list=[path_tb_pex5_egfp],
          7   model=model_keras,
    
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/correction.py in correct_experimental_traces_from_ptu_by_unet_prediction(path_list, model, pred_thresh, photon_count_bin, ntraces, save_as_csv)
        405             print('Processing correlation with correction by prediction '
        406                   'of dataset {}'.format(i + 1))
    --> 407             data['{}-pred'.format(i)] = correct_correlation_by_unet_prediction(
        408                 ntraces=ntraces,
        409                 traces_of_interest=ptu_1ms.astype(np.float64),
    
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/correction.py in correct_correlation_by_unet_prediction(ntraces, traces_of_interest, model, pred_thresh, fwhm, length_delimiter, traces_for_correlation, bin_for_correlation, verbose)
        215             ntraces_index=ntraces_index)
        216
    --> 217         predictions = model.predict(features_prepro, verbose=0)
        218         predictions = predictions.flatten()
        219
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py in predict(self, x, batch_size, verbose, steps, callbacks, max_queue_size, workers, use_multiprocessing)
       1661           for step in data_handler.steps():
       1662             callbacks.on_predict_batch_begin(step)
    -> 1663             tmp_batch_outputs = self.predict_function(iterator)
       1664             if data_handler.should_sync:
       1665               context.async_wait()
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
        816     tracing_count = self.experimental_get_tracing_count()
        817     with trace.Trace(self._name) as tm:
    --> 818       result = self._call(*args, **kwds)
        819       compiler = "xla" if self._jit_compile else "nonXla"
        820       new_tracing_count = self.experimental_get_tracing_count()
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
        860       # This is the first call of __call__, so we have to initialize.
        861       initializers = []
    --> 862       self._initialize(args, kwds, add_initializers_to=initializers)
        863     finally:
        864       # At this point we know that the initialization is complete (or less
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
        701     self._graph_deleter = FunctionDeleter(self._lifted_initializer_graph)
        702     self._concrete_stateful_fn = (
    --> 703         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
        704             *args, **kwds))
        705
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
       3018       args, kwargs = None, None
       3019     with self._lock:
    -> 3020       graph_function, _ = self._maybe_define_function(args, kwargs)
       3021     return graph_function
       3022
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
       3412
       3413           self._function_cache.missed.add(call_context_key)
    -> 3414           graph_function = self._create_graph_function(args, kwargs)
       3415           self._function_cache.primary[cache_key] = graph_function
       3416
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
       3247     arg_names = base_arg_names + missing_arg_names
       3248     graph_function = ConcreteFunction(
    -> 3249         func_graph_module.func_graph_from_py_func(
       3250             self._name,
       3251             self._python_function,
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
        996         _, original_func = tf_decorator.unwrap(python_func)
        997
    --> 998       func_outputs = python_func(*func_args, **func_kwargs)
        999
       1000       # invariant: `func_outputs` contains only Tensors, CompositeTensors,
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
        610             xla_context.Exit()
        611         else:
    --> 612           out = weak_wrapped_fn().__wrapped__(*args, **kwds)
        613         return out
        614
    
     ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
        983           except Exception as e:  # pylint:disable=broad-except
        984             if hasattr(e, "ag_error_metadata"):
    --> 985               raise e.ag_error_metadata.to_exception(e)
        986             else:
        987               raise
    
    ValueError: in user code:
    
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1512 predict_function  *
            return step_function(self, iterator)
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1502 step_function  **
            outputs = model.distribute_strategy.run(run_step, args=(data,))
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:1262 run
            return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:2734 call_for_each_replica
            return self._call_for_each_replica(fn, args, kwargs)
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:3423 _call_for_each_replica
            return fn(*args, **kwargs)
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1495 run_step  **
            outputs = model.predict_step(data)
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1468 predict_step
            return self(x, training=False)
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py:1018 __call__
            input_spec.assert_input_compatibility(self.input_spec, inputs, self.name)
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/input_spec.py:271 assert_input_compatibility
            raise ValueError('Input ' + str(input_index) +
    
        ValueError: Input 0 is incompatible with layer model: expected shape=(None, 16384, 1), found shape=(None, 8192, 1)
    

    So how do we solve this? I did not find the real cause, but a workaround is, to let the loaded model first predict a dummy input with the training size:

    test_features = np.zeros((2**14))
    test_features = np.reshape(test_features, (1, -1, 1))
    print(test_features.shape)
    predictions = model_keras.predict(test_features, verbose=0)
    predictions = predictions.flatten()
    predictions
    
    (1, 16384, 1)
    array([2.7520498e-08, 7.7272375e-09, 5.2612198e-10, ..., 9.8416812e-08,
           8.9488267e-08, 3.5880134e-07], dtype=float32)
    
  2. Test the algorithm: here it can be seen, that in the ideal case, the metadata from the single files is joined to each computation. For the computation of all traces, this will probably not be possible, because of traces, which get omitted before correlation, because their trace lengths shrank below 32 time steps after correction. This small traces can't be handled by multipletau.
    path_tb_pex5_egfp = '/beegfs/ye53nis/data/Pablo_structured_experiment'
    pred_thresh = [0.1, 0.3, 0.5, 0.7, 0.9]
    length_delimiter = 2**13  # for U-Net
    bin_for_correlation = 1e5
    out = correction.correct_experimental_traces_from_ptu_by_unet_prediction(
      path_list=path_tb_pex5_egfp,
      model=model_keras,
      pred_thresh=pred_thresh,
      photon_count_bin=bin_for_correlation,
      ntraces=None,
      save_as_csv=True)
    out
    
    Loading dataset 1 from path /beegfs/ye53nis/data/Pablo_structured_experiment with bin=1e6. This can take a while...
    1 of 2: /beegfs/ye53nis/data/Pablo_structured_experiment/DiO LUV 10uM in 20 nM AF48822_T273s_1.ptu
    2 of 2: /beegfs/ye53nis/data/Pablo_structured_experiment/DiO LUV 10uM in 20 nM AF48821_T260s_1.ptu
    Different binning was chosen for correlation. Loading dataset 1 with bin=100000.0. This can take a while...
    Processing correlation of unprocessed dataset 1
    Processing correlation with correction by prediction of dataset 1
    
      \(D\) in \(\frac{{\mu m\^2}}{{s}}\) \(\tau\_{{D}}\) in \(ms\) Trace lengths folder\id-traces\used Photon count bin for correlation in \(ns\) File\GUID File\CreatingTime Measurement\SubMode File\Comment TTResult\StopReason MeasDesc\GlobalResolution TTResult\NumberOfRecords MeasDesc\AcquisitionTime TTResult\MDescWarningFlags TTResult\StopAfter TTResultFormat\TTTRRecType TTResultFormat\BitsPerRecord UsrPowerDiode Header\End Number of Channels
    0 0.87156 12.932046 8192.0 0-orig 100000.0 {1FADBD63-E853-4886-82EB-0D4FEB0A2641} (2019, 10, 31, 11, 17, 29, 3, 304, 0) 0   0 0.0 6124808 10000 0 10000 16843524 32 1002.253351 <empty Tag> 1
    1 0.091734 122.866229 8192.0 0-orig 100000.0 {1104778F-509F-40AB-9176-580B3A1CE38D} (2019, 10, 31, 11, 17, 16, 3, 304, 0) 0   0 0.0 5881858 10000 0 10000 16843524 32 979.946455 <empty Tag> 1
    2 3.181256 3.542957 6413.0 0-pred-0.1 100000.0 {1FADBD63-E853-4886-82EB-0D4FEB0A2641} (2019, 10, 31, 11, 17, 29, 3, 304, 0) 0   0 0.0 6124808 10000 0 10000 16843524 32 1002.253351 <empty Tag> 1
    3 1.968434 5.725899 7132.0 0-pred-0.1 100000.0 {1104778F-509F-40AB-9176-580B3A1CE38D} (2019, 10, 31, 11, 17, 16, 3, 304, 0) 0   0 0.0 5881858 10000 0 10000 16843524 32 979.946455 <empty Tag> 1
    4 0.534267 21.096303 6903.0 0-pred-0.3 100000.0 {1FADBD63-E853-4886-82EB-0D4FEB0A2641} (2019, 10, 31, 11, 17, 29, 3, 304, 0) 0   0 0.0 6124808 10000 0 10000 16843524 32 1002.253351 <empty Tag> 1
    5 0.760655 14.817573 7490.0 0-pred-0.3 100000.0 {1104778F-509F-40AB-9176-580B3A1CE38D} (2019, 10, 31, 11, 17, 16, 3, 304, 0) 0   0 0.0 5881858 10000 0 10000 16843524 32 979.946455 <empty Tag> 1
    6 0.219843 51.268582 7178.0 0-pred-0.5 100000.0 {1FADBD63-E853-4886-82EB-0D4FEB0A2641} (2019, 10, 31, 11, 17, 29, 3, 304, 0) 0   0 0.0 6124808 10000 0 10000 16843524 32 1002.253351 <empty Tag> 1
    7 0.453877 24.832866 7659.0 0-pred-0.5 100000.0 {1104778F-509F-40AB-9176-580B3A1CE38D} (2019, 10, 31, 11, 17, 16, 3, 304, 0) 0   0 0.0 5881858 10000 0 10000 16843524 32 979.946455 <empty Tag> 1
    8 0.144134 78.198714 7407.0 0-pred-0.7 100000.0 {1FADBD63-E853-4886-82EB-0D4FEB0A2641} (2019, 10, 31, 11, 17, 29, 3, 304, 0) 0   0 0.0 6124808 10000 0 10000 16843524 32 1002.253351 <empty Tag> 1
    9 0.350758 32.133387 7785.0 0-pred-0.7 100000.0 {1104778F-509F-40AB-9176-580B3A1CE38D} (2019, 10, 31, 11, 17, 16, 3, 304, 0) 0   0 0.0 5881858 10000 0 10000 16843524 32 979.946455 <empty Tag> 1
    10 0.092882 121.348056 7692.0 0-pred-0.9 100000.0 {1FADBD63-E853-4886-82EB-0D4FEB0A2641} (2019, 10, 31, 11, 17, 29, 3, 304, 0) 0   0 0.0 6124808 10000 0 10000 16843524 32 1002.253351 <empty Tag> 1
    11 0.281531 40.03487 7937.0 0-pred-0.9 100000.0 {1104778F-509F-40AB-9176-580B3A1CE38D} (2019, 10, 31, 11, 17, 16, 3, 304, 0) 0   0 0.0 5881858 10000 0 10000 16843524 32 979.946455 <empty Tag> 1

    12 rows × 94 columns

  3. Run correction for all experimental traces
    path_pex5_exp = ['/beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu', '/beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu']
    pred_thresh = [0.1, 0.3, 0.5, 0.7, 0.9]
    length_delimiter = 2**13  # for U-Net
    bin_for_correlation = 1e5
    out = correction.correct_experimental_traces_from_ptu_by_unet_prediction(
      path_list=path_pex5_exp,
      model=model_keras,
      pred_thresh=pred_thresh,
      photon_count_bin=bin_for_correlation,
      ntraces=400,
      save_as_csv=True)
    out
    
    Loading dataset 1 from path /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu with bin=1e6. This can take a while...
    1 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48891_T1082s_1.ptu
    2 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488129_T1537s_1.ptu
    3 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488210_T2505s_1.ptu
    4 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488199_T2375s_1.ptu
    5 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488171_T2040s_1.ptu
    6 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488229_T2733s_1.ptu
    7 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48890_T1070s_1.ptu
    8 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488113_T1352s_1.ptu
    9 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48883_T986s_1.ptu10 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488227_T2709s_1.ptu
    11 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488174_T2088s_1.ptu
    12 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488102_T1214s_1.ptu
    13 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488131_T1561s_1.ptu
    14 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48826_T302s_1.ptu
    15 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488162_T1932s_1.ptu
    16 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488190_T2267s_1.ptu
    17 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488189_T2255s_1.ptu
    18 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488208_T2481s_1.ptu
    19 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48827_T314s_1.ptu
    20 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488221_T2637s_1.ptu
    21 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488153_T1834s_1.ptu
    22 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488157_T1883s_1.ptu
    23 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488197_T2351s_1.ptu
    24 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48822_T254s_1.ptu
    25 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48816_T182s_1.ptu
    26 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488179_T2148s_1.ptu
    27 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488104_T1238s_1.ptu
    28 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48870_T833s_1.ptu
    29 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48857_T676s_1.ptu
    30 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488125_T1496s_1.ptu
    31 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48839_T459s_1.ptu
    32 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488135_T1610s_1.ptu
    33 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48850_T592s_1.ptu
    34 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488159_T1907s_1.ptu
    35 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48851_T604s_1.ptu
    36 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488158_T1895s_1.ptu
    37 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48878_T926s_1.ptu
    38 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48838_T447s_1.ptu
    39 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48869_T821s_1.ptu
    40 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48834_T399s_1.ptu
    41 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48843_T506s_1.ptu
    42 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48890_T1076s_1.ptu
    43 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488184_T2209s_1.ptu
    44 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48863_T748s_1.ptu
    45 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488103_T1232s_1.ptu
    46 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488181_T2159s_1.ptu
    47 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48859_T698s_1.ptu
    48 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48872_T855s_1.ptu
    49 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488150_T1789s_1.ptu
    50 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488241_T2877s_1.ptu
    51 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488204_T2434s_1.ptu
    52 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48877_T918s_1.ptu
    53 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488180_T2147s_1.ptu
    54 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488183_T2197s_1.ptu
    55 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488160_T1908s_1.ptu
    56 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488238_T2842s_1.ptu
    57 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488165_T1968s_1.ptu
    58 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488234_T2793s_1.ptu
    59 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48882_T979s_1.ptu
    60 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48896_T1148s_1.ptu
    61 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488107_T1280s_1.ptu
    62 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488138_T1654s_1.ptu
    63 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488124_T1484s_1.ptu
    64 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488187_T2245s_1.ptu
    65 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48874_T879s_1.ptu
    66 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488159_T1896s_1.ptu
    67 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488177_T2124s_1.ptu
    68 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488111_T1328s_1.ptu
    69 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48873_T867s_1.ptu
    70 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48881_T962s_1.ptu
    71 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48853_T628s_1.ptu
    72 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48842_T496s_1.ptu
    73 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48850_T590s_1.ptu
    74 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48899_T1184s_1.ptu
    75 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488196_T2339s_1.ptu
    76 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488164_T1956s_1.ptu
    77 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488217_T2589s_1.ptu
    78 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488218_T2601s_1.ptu
    79 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488119_T1418s_1.ptu
    80 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488206_T2458s_1.ptu
    81 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48866_T785s_1.ptu
    82 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488103_T1226s_1.ptu
    83 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48824_T278s_1.ptu
    84 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4887_T74s_1.ptu
    85 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488176_T2099s_1.ptu
    86 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48817_T194s_1.ptu
    87 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488113_T1346s_1.ptu
    88 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48858_T688s_1.ptu
    89 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488211_T2517s_1.ptu
    90 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48860_T712s_1.ptu
    91 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48833_T387s_1.ptu
    92 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48858_T686s_1.ptu
    93 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48813_T146s_1.ptu
    94 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488126_T1502s_1.ptu
    95 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48894_T1118s_1.ptu
    96 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488110_T1316s_1.ptu
    97 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488188_T2243s_1.ptu
    98 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488120_T1430s_1.ptu
    99 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488219_T2613s_1.ptu
    100 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488138_T1646s_1.ptu
    101 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488187_T2231s_1.ptu
    102 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488146_T1742s_1.ptu
    103 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488155_T1859s_1.ptu
    104 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488174_T2075s_1.ptu
    105 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488158_T1884s_1.ptu
    106 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488244_T2914s_1.ptu
    107 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4886_T62s_1.ptu
    108 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48885_T1015s_1.ptu
    109 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488165_T1979s_1.ptu
    110 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48857_T674s_1.ptu
    111 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488191_T2279s_1.ptu
    112 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488183_T2183s_1.ptu
    113 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488141_T1689s_1.ptu
    114 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488189_T2269s_1.ptu
    115 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48823_T266s_1.ptu
    116 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488209_T2493s_1.ptu
    117 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488136_T1630s_1.ptu
    118 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488185_T2221s_1.ptu
    119 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488179_T2135s_1.ptu
    120 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48866_T782s_1.ptu
    121 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488123_T1472s_1.ptu
    122 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488160_T1919s_1.ptu
    123 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48856_T664s_1.ptu
    124 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48830_T351s_1.ptu
    125 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48832_T375s_1.ptu
    126 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48859_T700s_1.ptu
    127 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488129_T1545s_1.ptu
    128 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488143_T1714s_1.ptu
    129 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48886_T1022s_1.ptu
    130 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488147_T1754s_1.ptu
    131 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488193_T2303s_1.ptu
    132 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488105_T1250s_1.ptu
    133 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488228_T2721s_1.ptu
    134 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48891_T1088s_1.ptu
    135 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488170_T2040s_1.ptu
    136 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48835_T411s_1.ptu
    137 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48888_T1046s_1.ptu
    138 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488123_T1466s_1.ptu
    139 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48876_T906s_1.ptu
    140 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48822_T255s_1.ptu
    141 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488145_T1730s_1.ptu
    142 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488114_T1358s_1.ptu
    143 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48824_T279s_1.ptu
    144 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48854_T638s_1.ptu
    145 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488247_T2949s_1.ptu
    146 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48844_T519s_1.ptu
    147 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48841_T482s_1.ptu
    148 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48828_T327s_1.ptu
    149 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48840_T471s_1.ptu
    150 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488225_T2685s_1.ptu
    151 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48831_T362s_1.ptu
    152 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48849_T578s_1.ptu
    153 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48861_T722s_1.ptu
    154 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48852_T614s_1.ptu
    155 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48852_T616s_1.ptu
    156 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488149_T1778s_1.ptu
    157 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488169_T2028s_1.ptu
    158 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4883_T26s_1.ptu
    159 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48881_T966s_1.ptu
    160 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48837_T435s_1.ptu
    161 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488139_T1666s_1.ptu
    162 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488194_T2315s_1.ptu
    163 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488205_T2446s_1.ptu
    164 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48818_T206s_1.ptu
    165 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488128_T1533s_1.ptu
    166 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48851_T602s_1.ptu
    167 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488116_T1388s_1.ptu
    168 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488184_T2196s_1.ptu
    169 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488117_T1394s_1.ptu
    170 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48869_T818s_1.ptu
    171 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48895_T1130s_1.ptu
    172 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488173_T2076s_1.ptu
    173 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488106_T1262s_1.ptu
    174 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488249_T2993s_1.ptu
    175 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48889_T1063s_1.ptu
    176 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48845_T531s_1.ptu
    177 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488117_T1400s_1.ptu
    178 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488192_T2291s_1.ptu
    179 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488166_T1992s_1.ptu
    180 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48896_T1142s_1.ptu
    181 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48894_T1124s_1.ptu
    182 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488177_T2111s_1.ptu
    183 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48862_T736s_1.ptu
    184 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488118_T1412s_1.ptu
    185 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48892_T1100s_1.ptu
    186 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48847_T555s_1.ptu
    187 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488223_T2661s_1.ptu
    188 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48853_T626s_1.ptu
    189 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488136_T1622s_1.ptu
    190 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488151_T1810s_1.ptu
    191 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488245_T2925s_1.ptu
    192 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48879_T942s_1.ptu
    193 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488121_T1448s_1.ptu
    194 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48868_T809s_1.ptu
    195 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488201_T2398s_1.ptu
    196 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488140_T1670s_1.ptu
    197 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488250_T3006s_1.ptu
    198 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4884_T38s_1.ptu
    199 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488127_T1514s_1.ptu
    200 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488104_T1244s_1.ptu
    201 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488181_T2172s_1.ptu
    202 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488115_T1370s_1.ptu
    203 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488140_T1678s_1.ptu
    204 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48855_T652s_1.ptu
    205 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48884_T1003s_1.ptu
    206 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48846_T544s_1.ptu
    207 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488131_T1569s_1.ptu
    208 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488236_T2817s_1.ptu
    209 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488178_T2123s_1.ptu
    210 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48873_T869s_1.ptu
    211 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488164_T1968s_1.ptu
    212 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488249_T2973s_1.ptu
    213 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488151_T1801s_1.ptu
    214 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488235_T2805s_1.ptu
    215 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488170_T2028s_1.ptu
    216 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488180_T2160s_1.ptu
    217 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488135_T1618s_1.ptu
    218 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488190_T2281s_1.ptu
    219 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48864_T758s_1.ptu
    220 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48867_T794s_1.ptu
    221 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48875_T891s_1.ptu
    222 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48871_T842s_1.ptu
    223 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488213_T2541s_1.ptu
    224 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488182_T2184s_1.ptu
    225 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48841_T484s_1.ptu
    226 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488171_T2052s_1.ptu
    227 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488153_T1825s_1.ptu
    228 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488132_T1581s_1.ptu
    229 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488147_T1762s_1.ptu
    230 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488110_T1310s_1.ptu
    231 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48885_T1010s_1.ptu
    232 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48825_T290s_1.ptu
    233 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488168_T2004s_1.ptu
    234 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48820_T230s_1.ptu
    235 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488162_T1944s_1.ptu
    236 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488203_T2422s_1.ptu
    237 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488152_T1822s_1.ptu
    238 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48821_T243s_1.ptu
    239 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488167_T2004s_1.ptu
    240 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48837_T436s_1.ptu
    241 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488166_T1980s_1.ptu
    242 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488200_T2386s_1.ptu
    243 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488122_T1454s_1.ptu
    244 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48843_T508s_1.ptu
    245 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488102_T1220s_1.ptu
    246 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48865_T773s_1.ptu
    247 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488161_T1920s_1.ptu
    248 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48823_T267s_1.ptu
    249 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48863_T746s_1.ptu
    250 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488132_T1573s_1.ptu
    251 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48867_T797s_1.ptu
    252 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488144_T1726s_1.ptu
    253 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488145_T1738s_1.ptu
    254 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488169_T2016s_1.ptu
    255 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48889_T1058s_1.ptu
    256 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48860_T710s_1.ptu
    257 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48831_T363s_1.ptu
    258 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488215_T2565s_1.ptu
    259 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48878_T930s_1.ptu
    260 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48865_T770s_1.ptu
    261 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48870_T830s_1.ptu
    262 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488114_T1364s_1.ptu
    263 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488148_T1774s_1.ptu
    264 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488224_T2673s_1.ptu
    265 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488172_T2052s_1.ptu
    266 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488167_T1992s_1.ptu
    267 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488100_T1190s_1.ptu
    268 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48821_T242s_1.ptu
    269 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488240_T2865s_1.ptu
    270 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488142_T1701s_1.ptu
    271 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488130_T1549s_1.ptu
    272 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48844_T520s_1.ptu
    273 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48861_T724s_1.ptu
    274 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48814_T158s_1.ptu
    275 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488163_T1956s_1.ptu
    276 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488173_T2064s_1.ptu
    277 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4885_T50s_1.ptu
    278 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48849_T580s_1.ptu
    279 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48840_T472s_1.ptu
    280 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48883_T991s_1.ptu
    281 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488247_T2969s_1.ptu
    282 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488150_T1798s_1.ptu
    283 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48848_T566s_1.ptu
    284 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48897_T1160s_1.ptu
    285 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488118_T1406s_1.ptu
    286 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4889_T98s_1.ptu
    287 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48812_T134s_1.ptu
    288 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488222_T2649s_1.ptu
    289 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48887_T1039s_1.ptu
    290 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48879_T938s_1.ptu
    291 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48888_T1051s_1.ptu
    292 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48876_T902s_1.ptu
    293 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48819_T218s_1.ptu
    294 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48893_T1112s_1.ptu
    295 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488121_T1442s_1.ptu
    296 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488101_T1208s_1.ptu
    297 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488109_T1304s_1.ptu
    298 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488128_T1526s_1.ptu
    299 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488207_T2470s_1.ptu
    300 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488246_T2937s_1.ptu
    301 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488242_T2890s_1.ptu
    302 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48854_T640s_1.ptu
    303 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48847_T554s_1.ptu
    304 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488108_T1292s_1.ptu
    305 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488134_T1597s_1.ptu
    306 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4888_T86s_1.ptu
    307 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48872_T857s_1.ptu
    308 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488149_T1786s_1.ptu
    309 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48825_T291s_1.ptu
    310 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488182_T2171s_1.ptu
    311 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488156_T1871s_1.ptu
    312 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488220_T2624s_1.ptu
    313 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488248_T2981s_1.ptu
    314 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48897_T1154s_1.ptu
    315 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48848_T568s_1.ptu
    316 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488175_T2100s_1.ptu
    317 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488154_T1847s_1.ptu
    318 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488245_T2945s_1.ptu
    319 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488139_T1658s_1.ptu
    320 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48830_T350s_1.ptu
    321 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488175_T2087s_1.ptu
    322 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48856_T662s_1.ptu
    323 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488230_T2745s_1.ptu
    324 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48875_T894s_1.ptu
    325 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488127_T1521s_1.ptu
    326 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488195_T2327s_1.ptu
    327 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48884_T998s_1.ptu
    328 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488186_T2219s_1.ptu
    329 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48882_T974s_1.ptu
    330 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488134_T1606s_1.ptu
    331 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488108_T1286s_1.ptu
    332 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48827_T315s_1.ptu
    333 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488214_T2553s_1.ptu
    334 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488163_T1944s_1.ptu
    335 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48839_T460s_1.ptu
    336 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488186_T2233s_1.ptu
    337 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488243_T2902s_1.ptu
    338 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488137_T1642s_1.ptu
    339 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488156_T1861s_1.ptu
    340 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4882_T14s_1.ptu
    341 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488142_T1694s_1.ptu
    342 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48811_T123s_1.ptu
    343 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488248_T2961s_1.ptu
    344 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48895_T1136s_1.ptu
    345 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48826_T303s_1.ptu
    346 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488191_T2293s_1.ptu
    347 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488137_T1634s_1.ptu
    348 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488133_T1585s_1.ptu
    349 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488216_T2577s_1.ptu
    350 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488107_T1274s_1.ptu
    351 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488130_T1557s_1.ptu
    352 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488120_T1436s_1.ptu
    353 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48868_T806s_1.ptu
    354 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488125_T1490s_1.ptu
    355 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48832_T374s_1.ptu
    356 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488119_T1424s_1.ptu
    357 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48877_T914s_1.ptu
    358 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488188_T2257s_1.ptu
    359 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48836_T423s_1.ptu
    360 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488176_T2112s_1.ptu
    361 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488112_T1334s_1.ptu
    362 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48815_T170s_1.ptu
    363 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48838_T448s_1.ptu
    364 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488178_T2136s_1.ptu
    365 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488198_T2363s_1.ptu
    366 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48855_T650s_1.ptu
    367 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48899_T1178s_1.ptu
    368 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488161_T1931s_1.ptu
    369 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488246_T2957s_1.ptu
    370 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488231_T2757s_1.ptu
    371 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488122_T1460s_1.ptu
    372 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488157_T1873s_1.ptu
    373 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488194_T2329s_1.ptu
    374 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48846_T542s_1.ptu
    375 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488144_T1718s_1.ptu
    376 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488111_T1322s_1.ptu
    377 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488143_T1705s_1.ptu
    378 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48874_T882s_1.ptu
    379 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488124_T1478s_1.ptu
    380 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488106_T1268s_1.ptu
    381 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48892_T1094s_1.ptu
    382 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488115_T1376s_1.ptu
    383 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48864_T761s_1.ptu
    384 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488239_T2853s_1.ptu
    385 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48893_T1106s_1.ptu
    386 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488154_T1837s_1.ptu
    387 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488105_T1256s_1.ptu
    388 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48829_T338s_1.ptu
    389 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48871_T845s_1.ptu
    390 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48898_T1172s_1.ptu
    391 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48880_T954s_1.ptu
    392 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48811_T122s_1.ptu
    393 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488141_T1682s_1.ptu
    394 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488237_T2829s_1.ptu
    395 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48810_T110s_1.ptu
    396 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48898_T1166s_1.ptu
    397 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48812_T135s_1.ptu
    398 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48887_T1034s_1.ptu
    399 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488202_T2410s_1.ptu
    400 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48828_T326s_1.ptu
    401 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48886_T1027s_1.ptu
    Different binning was chosen for correlation. Loading dataset 1 with bin=100000.0. This can take a while...
    Processing correlation of unprocessed dataset 1
    Processing correlation with correction by prediction of dataset 1
    Loading dataset 2 from path /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu with bin=1e6. This can take a while...
    1 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48869_T825s_1.ptu
    2 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488196_T2359s_1.ptu
    3 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488146_T1755s_1.ptu
    4 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488227_T2733s_1.ptu
    5 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48823_T287s_1.ptu
    6 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48829_T342s_1.ptu
    7 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488194_T2334s_1.ptu
    8 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48851_T607s_1.ptu
    9 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488210_T2528s_1.ptu
    10 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488234_T2818s_1.ptu
    11 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48882_T982s_1.ptu
    12 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488251_T3022s_1.ptu
    13 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48848_T610s_1.ptu
    14 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488160_T1924s_1.ptu
    15 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48846_T584s_1.ptu
    16 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488268_T3227s_1.ptu
    17 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48823_T268s_1.ptu
    18 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48832_T378s_1.ptu
    19 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488241_T2902s_1.ptu
    20 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488198_T2383s_1.ptu
    21 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48869_T823s_1.ptu
    22 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48895_T1139s_1.ptu
    23 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48878_T934s_1.ptu
    24 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488100_T1197s_1.ptu
    25 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488155_T1864s_1.ptu
    26 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488132_T1585s_1.ptu
    27 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48866_T787s_1.ptu
    28 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48877_T922s_1.ptu
    29 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48861_T728s_1.ptu
    30 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488129_T1548s_1.ptu
    31 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48825_T292s_1.ptu
    32 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488205_T2468s_1.ptu
    33 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4889_T106s_1.ptu
    34 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48840_T474s_1.ptu
    35 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48843_T546s_1.ptu
    36 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48887_T1042s_1.ptu
    37 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488116_T1391s_1.ptu
    38 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4884_T39s_1.ptu
    39 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488110_T1318s_1.ptu
    40 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488218_T2625s_1.ptu
    41 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48870_T834s_1.ptu
    42 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488270_T3251s_1.ptu
    43 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488177_T2129s_1.ptu
    44 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488136_T1633s_1.ptu
    45 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48844_T521s_1.ptu
    46 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48829_T341s_1.ptu
    47 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4887_T75s_1.ptu
    48 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48842_T497s_1.ptu
    49 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488291_T3505s_1.ptu
    50 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488117_T1403s_1.ptu
    51 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48814_T159s_1.ptu
    52 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488215_T2589s_1.ptu
    53 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488128_T1536s_1.ptu
    54 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488135_T1621s_1.ptu
    55 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48870_T837s_1.ptu
    56 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488239_T2878s_1.ptu
    57 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48827_T317s_1.ptu
    58 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488231_T2781s_1.ptu
    59 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488294_T3541s_1.ptu
    60 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488247_T2974s_1.ptu
    61 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488226_T2721s_1.ptu
    62 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488182_T2189s_1.ptu
    63 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488248_T2986s_1.ptu
    64 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488216_T2601s_1.ptu
    65 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48824_T282s_1.ptu
    66 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488152_T1827s_1.ptu
    67 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48868_T811s_1.ptu
    68 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488112_T1342s_1.ptu
    69 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48811_T123s_1.ptu
    70 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488185_T2225s_1.ptu
    71 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48828_T330s_1.ptu
    72 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488232_T2793s_1.ptu
    73 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488293_T3529s_1.ptu
    74 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48825_T294s_1.ptu
    75 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4886_T66s_1.ptu
    76 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48822_T273s_1.ptu
    77 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48889_T1067s_1.ptu
    78 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48816_T183s_1.ptu
    79 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48879_T946s_1.ptu
    80 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48876_T910s_1.ptu
    81 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48840_T507s_1.ptu
    82 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48849_T583s_1.ptu
    83 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48850_T595s_1.ptu
    84 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48822_T257s_1.ptu
    85 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488184_T2213s_1.ptu
    86 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48813_T148s_1.ptu
    87 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488121_T1452s_1.ptu
    88 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488254_T3059s_1.ptu
    89 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48814_T171s_1.ptu
    90 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48849_T581s_1.ptu
    91 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488122_T1464s_1.ptu
    92 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488153_T1839s_1.ptu
    93 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48851_T605s_1.ptu
    94 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48819_T222s_1.ptu
    95 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488104_T1246s_1.ptu
    96 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488179_T2153s_1.ptu
    97 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488130_T1560s_1.ptu
    98 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48874_T885s_1.ptu
    99 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488115_T1379s_1.ptu
    100 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488282_T3397s_1.ptu
    101 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488259_T3119s_1.ptu
    102 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48858_T690s_1.ptu
    103 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48821_T244s_1.ptu
    104 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48825_T313s_1.ptu
    105 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48861_T726s_1.ptu
    106 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48817_T196s_1.ptu
    107 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488265_T3191s_1.ptu
    108 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48834_T400s_1.ptu
    109 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48854_T643s_1.ptu
    110 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48873_T873s_1.ptu
    111 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48857_T678s_1.ptu
    112 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488208_T2504s_1.ptu
    113 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4884_T41s_1.ptu
    114 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48810_T120s_1.ptu
    115 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48873_T870s_1.ptu
    116 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48872_T859s_1.ptu
    117 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488180_T2165s_1.ptu
    118 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48828_T328s_1.ptu
    119 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488224_T2697s_1.ptu
    120 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488167_T2009s_1.ptu
    121 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488237_T2854s_1.ptu
    122 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488258_T3107s_1.ptu
    123 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488222_T2673s_1.ptu
    124 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48845_T534s_1.ptu
    125 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488280_T3373s_1.ptu
    126 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48848_T569s_1.ptu
    127 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48812_T145s_1.ptu
    128 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48844_T522s_1.ptu
    129 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48828_T351s_1.ptu
    130 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488173_T2081s_1.ptu
    131 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48893_T1115s_1.ptu
    132 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48865_T774s_1.ptu
    133 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48862_T740s_1.ptu
    134 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488187_T2250s_1.ptu
    135 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48812_T136s_1.ptu
    136 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48853_T631s_1.ptu
    137 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48843_T509s_1.ptu
    138 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48880_T955s_1.ptu
    139 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48845_T535s_1.ptu
    140 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488133_T1597s_1.ptu
    141 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48836_T426s_1.ptu
    142 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488138_T1658s_1.ptu
    143 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488235_T2830s_1.ptu
    144 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488111_T1330s_1.ptu
    145 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488156_T1876s_1.ptu
    146 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48855_T655s_1.ptu
    147 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4887_T80s_1.ptu
    148 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48831_T365s_1.ptu
    149 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48854_T642s_1.ptu
    150 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48872_T861s_1.ptu
    151 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488107_T1282s_1.ptu
    152 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48824_T280s_1.ptu
    153 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488256_T3083s_1.ptu
    154 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488204_T2456s_1.ptu
    155 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488197_T2371s_1.ptu
    156 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48852_T619s_1.ptu
    157 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488176_T2117s_1.ptu
    158 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488229_T2757s_1.ptu
    159 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48842_T498s_1.ptu
    160 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48826_T305s_1.ptu
    161 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48864_T762s_1.ptu
    162 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488289_T3481s_1.ptu
    163 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48849_T624s_1.ptu
    164 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488249_T2998s_1.ptu
    165 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48816_T184s_1.ptu
    166 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48817_T197s_1.ptu
    167 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48884_T1006s_1.ptu
    168 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488178_T2141s_1.ptu
    169 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48855_T654s_1.ptu
    170 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48813_T158s_1.ptu
    171 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488118_T1415s_1.ptu
    172 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488127_T1524s_1.ptu
    173 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48836_T425s_1.ptu
    174 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48830_T352s_1.ptu
    175 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48829_T340s_1.ptu
    176 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488261_T3143s_1.ptu
    177 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48837_T468s_1.ptu
    178 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48830_T354s_1.ptu
    179 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48819_T220s_1.ptu
    180 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488236_T2842s_1.ptu
    181 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488257_T3095s_1.ptu
    182 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48819_T219s_1.ptu
    183 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488253_T3047s_1.ptu
    184 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48841_T485s_1.ptu
    185 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48830_T377s_1.ptu
    186 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488201_T2420s_1.ptu
    187 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48835_T412s_1.ptu
    188 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488269_T3239s_1.ptu
    189 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48846_T545s_1.ptu
    190 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48833_T416s_1.ptu
    191 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48839_T461s_1.ptu
    192 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48850_T637s_1.ptu
    193 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488276_T3324s_1.ptu
    194 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488286_T3445s_1.ptu
    195 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488181_T2177s_1.ptu
    196 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488199_T2395s_1.ptu
    197 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488244_T2938s_1.ptu
    198 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488101_T1209s_1.ptu
    199 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48881_T970s_1.ptu
    200 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488126_T1512s_1.ptu
    201 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488223_T2685s_1.ptu
    202 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48847_T557s_1.ptu
    203 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488158_T1900s_1.ptu
    204 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488139_T1670s_1.ptu
    205 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48824_T281s_1.ptu
    206 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48859_T702s_1.ptu
    207 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48846_T546s_1.ptu
    208 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488267_T3215s_1.ptu
    209 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4883_T28s_1.ptu
    210 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48827_T318s_1.ptu
    211 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48820_T232s_1.ptu
    212 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48860_T716s_1.ptu
    213 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4882_T15s_1.ptu
    214 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48896_T1151s_1.ptu
    215 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48830_T353s_1.ptu
    216 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488262_T3155s_1.ptu
    217 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48899_T1187s_1.ptu
    218 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488209_T2516s_1.ptu
    219 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488150_T1803s_1.ptu
    220 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488147_T1767s_1.ptu
    221 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488134_T1609s_1.ptu
    222 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48815_T171s_1.ptu
    223 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488264_T3179s_1.ptu
    224 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488298_T3590s_1.ptu
    225 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488193_T2322s_1.ptu
    226 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4888_T87s_1.ptu
    227 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488106_T1270s_1.ptu
    228 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488284_T3421s_1.ptu
    229 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488145_T1743s_1.ptu
    230 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48828_T329s_1.ptu
    231 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488183_T2201s_1.ptu
    232 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48856_T668s_1.ptu
    233 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4888_T93s_1.ptu
    234 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4889_T100s_1.ptu
    235 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48818_T222s_1.ptu
    236 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48835_T442s_1.ptu
    237 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488175_T2105s_1.ptu
    238 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48841_T486s_1.ptu
    239 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48833_T390s_1.ptu
    240 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488245_T2950s_1.ptu
    241 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488195_T2347s_1.ptu
    242 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48867_T799s_1.ptu
    243 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48839_T494s_1.ptu
    244 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48817_T195s_1.ptu
    245 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488105_T1258s_1.ptu
    246 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48897_T1163s_1.ptu
    247 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48835_T413s_1.ptu
    248 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48820_T248s_1.ptu
    249 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48841_T520s_1.ptu
    250 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4881_T0s_1.ptu
    251 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488165_T1984s_1.ptu
    252 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48887_T1040s_1.ptu
    253 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48817_T210s_1.ptu
    254 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4883_T27s_1.ptu
    255 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48891_T1091s_1.ptu
    256 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488233_T2806s_1.ptu
    257 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48811_T132s_1.ptu
    258 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48876_T906s_1.ptu
    259 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48839_T462s_1.ptu
    260 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488109_T1306s_1.ptu
    261 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488162_T1948s_1.ptu
    262 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48858_T692s_1.ptu
    263 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488151_T1815s_1.ptu
    264 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488203_T2444s_1.ptu
    265 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48815_T184s_1.ptu
    266 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48831_T366s_1.ptu
    267 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48883_T991s_1.ptu
    268 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48881_T967s_1.ptu
    269 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488287_T3457s_1.ptu
    270 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48884_T1003s_1.ptu
    271 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48886_T1031s_1.ptu
    272 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48820_T234s_1.ptu
    273 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48845_T533s_1.ptu
    274 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488100_T1199s_1.ptu
    275 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488214_T2577s_1.ptu
    276 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48845_T571s_1.ptu
    277 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48856_T666s_1.ptu
    278 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488172_T2069s_1.ptu
    279 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48837_T438s_1.ptu
    280 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48822_T256s_1.ptu
    281 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488243_T2926s_1.ptu
    282 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488240_T2890s_1.ptu
    283 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488206_T2480s_1.ptu
    284 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48818_T208s_1.ptu
    285 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48866_T788s_1.ptu
    286 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48863_T752s_1.ptu
    287 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488238_T2866s_1.ptu
    288 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488260_T3131s_1.ptu
    289 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48818_T210s_1.ptu
    290 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48892_T1103s_1.ptu
    291 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48813_T147s_1.ptu
    292 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48834_T402s_1.ptu
    293 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48853_T630s_1.ptu
    294 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488114_T1367s_1.ptu
    295 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488300_T3614s_1.ptu
    296 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488119_T1427s_1.ptu
    297 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48868_T813s_1.ptu
    298 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488157_T1888s_1.ptu
    299 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488188_T2262s_1.ptu
    300 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488212_T2552s_1.ptu
    301 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48833_T388s_1.ptu
    302 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48864_T764s_1.ptu
    303 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48823_T269s_1.ptu
    304 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488252_T3035s_1.ptu
    305 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488169_T2032s_1.ptu
    306 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48890_T1079s_1.ptu
    307 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48871_T847s_1.ptu
    308 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488137_T1646s_1.ptu
    309 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488166_T1997s_1.ptu
    310 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488283_T3409s_1.ptu
    311 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488279_T3360s_1.ptu
    312 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488125_T1500s_1.ptu
    313 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488200_T2408s_1.ptu
    314 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48882_T979s_1.ptu
    315 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488281_T3385s_1.ptu
    316 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48811_T124s_1.ptu
    317 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488102_T1221s_1.ptu
    318 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48825_T293s_1.ptu
    319 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488186_T2237s_1.ptu
    320 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488274_T3299s_1.ptu
    321 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48838_T450s_1.ptu
    322 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48826_T304s_1.ptu
    323 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488163_T1960s_1.ptu
    324 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488174_T2093s_1.ptu
    325 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488250_T3010s_1.ptu
    326 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488255_T3071s_1.ptu
    327 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48847_T597s_1.ptu
    328 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488288_T3469s_1.ptu
    329 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488103_T1234s_1.ptu
    330 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488164_T1972s_1.ptu
    331 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48885_T1018s_1.ptu
    332 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48821_T246s_1.ptu
    333 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48812_T135s_1.ptu
    334 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488242_T2914s_1.ptu
    335 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488142_T1706s_1.ptu
    336 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488159_T1912s_1.ptu
    337 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4887_T76s_1.ptu
    338 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48886_T1028s_1.ptu
    339 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488263_T3167s_1.ptu
    340 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48824_T299s_1.ptu
    341 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488113_T1354s_1.ptu
    342 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488108_T1294s_1.ptu
    343 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48816_T197s_1.ptu
    344 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48863_T750s_1.ptu
    345 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488140_T1682s_1.ptu
    346 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488230_T2769s_1.ptu
    347 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488131_T1573s_1.ptu
    348 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488220_T2649s_1.ptu
    349 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48838_T449s_1.ptu
    350 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488154_T1852s_1.ptu
    351 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48838_T481s_1.ptu
    352 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488120_T1439s_1.ptu
    353 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488207_T2492s_1.ptu
    354 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488171_T2056s_1.ptu
    355 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488190_T2286s_1.ptu
    356 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48847_T558s_1.ptu
    357 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48898_T1175s_1.ptu
    358 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48862_T738s_1.ptu
    359 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48816_T186s_1.ptu
    360 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488299_T3602s_1.ptu
    361 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488219_T2637s_1.ptu
    362 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48810_T111s_1.ptu
    363 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488278_T3348s_1.ptu
    364 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488189_T2274s_1.ptu
    365 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48859_T704s_1.ptu
    366 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48832_T376s_1.ptu
    367 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48865_T776s_1.ptu
    368 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488225_T2709s_1.ptu
    369 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48871_T849s_1.ptu
    370 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48883_T994s_1.ptu
    371 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48831_T390s_1.ptu
    372 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488272_T3276s_1.ptu
    373 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488148_T1779s_1.ptu
    374 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48875_T898s_1.ptu
    375 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48842_T532s_1.ptu
    376 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488290_T3493s_1.ptu
    377 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48877_T918s_1.ptu
    378 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488297_T3578s_1.ptu
    379 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488191_T2298s_1.ptu
    380 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48814_T160s_1.ptu
    381 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48844_T558s_1.ptu
    382 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488221_T2661s_1.ptu
    383 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488211_T2540s_1.ptu
    384 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48821_T260s_1.ptu
    385 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488123_T1475s_1.ptu
    386 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48852_T617s_1.ptu
    387 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488149_T1791s_1.ptu
    388 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488141_T1694s_1.ptu
    389 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48827_T338s_1.ptu
    390 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488217_T2613s_1.ptu
    391 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48848_T571s_1.ptu
    392 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4886_T63s_1.ptu
    393 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48843_T510s_1.ptu
    394 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48810_T112s_1.ptu
    395 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488124_T1488s_1.ptu
    396 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488271_T3263s_1.ptu
    397 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48840_T473s_1.ptu
    398 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48826_T306s_1.ptu
    399 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4885_T54s_1.ptu
    400 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48827_T316s_1.ptu
    401 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48850_T593s_1.ptu
    Different binning was chosen for correlation. Loading dataset 2 with bin=100000.0. This can take a while...
    Processing correlation of unprocessed dataset 2
    Processing correlation with correction by prediction of dataset 2
    src/fluotracify/applications/correction.py:508: UserWarning: Metadata is not saved with data. Reason: the correlation algorithm failed for one or more traces which were shorter than 32 time steps after correction.Since metadata is loaded in the beginning, it is not sure, which correlation is missing to ensure proper joining of data and metadata.
      'correlation algorithm failed for one or more traces '
    
      \(D\) in \(\frac{{\mu m\^2}}{{s}}\) \(\tau\_{{D}}\) in \(ms\) Trace lengths folder\id-traces\used Photon count bin for correlation in \(ns\)
    0 19.078159 0.590783 8192.0 0-orig 100000.0
    1 20.314897 0.554817 8192.0 0-orig 100000.0
    2 21.007772 0.536518 8192.0 0-orig 100000.0
    3 22.716317 0.496166 8192.0 0-orig 100000.0
    4 24.089236 0.467888 8192.0 0-orig 100000.0
    4697 0.006707 1680.558197 7719.0 1-pred-0.9 100000.0
    4698 0.071281 158.122526 7592.0 1-pred-0.9 100000.0
    4699 1.228636 9.173635 7876.0 1-pred-0.9 100000.0
    4700 0.258008 43.684844 7830.0 1-pred-0.9 100000.0
    4701 0.436051 25.848023 7974.0 1-pred-0.9 100000.0

    4702 rows × 5 columns

2.3.4 learnings from run 1

  • while loading data, the subsample folder was loaded even though it should not have been → move it out of the main folder.
  • in the end, 3 folders would be good: Nov2020-train, Nov2020-test, Nov2020-subsample (subsample is used for quick prototyping)
  • also, I will need 3 .csv files per simulated diffusion rate, because I want to test the algo on each of the 3 different cluster speeds
  • I created firstartifact_Nov2020_test and firstartifact_Nov2020_train:
      0.01 0.1 1.0
    0.069 train   2, 3, 6, 7, 8 9
    0.08 train 7 2, 6, 8, 10 4, 9
    0.1 train 4, 6, 8, 9, 10   3, 7
    0.2 train 3 1, 4, 6, 8 9, 10
    0.4 train 4, 10 2, 3, 9 6, 7
    0.6 train 10 4, 5, 6, 7 1, 2
    1.0 train 10 4, 7, 9 1, 2, 8
    3.0 train 5, 6, 8 10 1, 3, 9
    10 train 3, 4, 8, 9 6, 7 10
    50 train 6 9, 10 4, 5, 7, 8
    0.069 test 5 1 10
    0.08 test 5 3 1
    0.1 test 2 5 1
    0.2 test 2 7 5
    0.4 test 8 1 5
    0.6 test 8 3 9
    1.0 test 6 3 5
    3.0 test 4 7 2
    10 test 2 1 5
    50 test 2 3 1
  • NOTE: I accidentally deleted set004 (0.01) from 0.069 test file. That means the only remaining simulated file with D=0.01 has to be taken for test (set009) and there is NO training set for 0.01 cluster speed at 0.069 molecule speed. Also, there is no training set for 0.1 clusters and 0.1 molecules (sad!), because there was only one simulated one.

2.3.5 Run 2 - full dataset

2.3.5.1 Record metadata, git log
  1. current directory
    %cd /beegfs/ye53nis/drmed-git
    
    /beegfs/ye53nis/drmed-git
    
  1. git log
    !git log -3
    
    commit fa530079f3d8bdbba1eaa4c6bd3eed6a24789d84
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Tue Mar 16 22:20:39 2021 +0100
    
        update mlproject for separate train and test files
    
    commit c9dd5f025bbd95e58f8f34796be36bcdb8c1a253
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Tue Mar 16 22:13:35 2021 +0100
    
        Update train for separate train and test paths
    
    commit a070d3b531725e0fb37688dde80e990083ccf1cc
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Mon Mar 15 21:48:01 2021 +0100
    
        Fix photon count bin metadata 2
    
  2. Metadata of environment
    No of CPUs in system: 72
    No of CPUs the current process can use: 24
    load average: (29.05, 36.29, 43.21)
    os.uname():  posix.uname_result(sysname='Linux', nodename='node260', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
    PID of process: 41375
    RAM total: 199G, RAM used: 7.3G, RAM free: 116G
    the current directory: /beegfs/ye53nis/drmed-git
    My disk usage:
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/sda1             50G  3.3G   47G   7% /
    devtmpfs              94G     0   94G   0% /dev
    tmpfs                 94G  133M   94G   1% /dev/shm
    tmpfs                 94G   19M   94G   1% /run
    tmpfs                 94G     0   94G   0% /sys/fs/cgroup
    nfs02-ib:/data01      88T   69T   19T  79% /data01
    nfs01-ib:/home        80T   66T   15T  82% /home
    nfs01-ib:/cluster    2.0T  417G  1.6T  21% /cluster
    nfs03-ib:/pool/work  100T   78T   23T  78% /nfsdata
    /dev/sda5            2.0G   66M  2.0G   4% /tmp
    /dev/sda6            169G  3.0G  166G   2% /local
    /dev/sda3            6.0G  397M  5.7G   7% /var
    beegfs_nodev         524T  303T  222T  58% /beegfs
    tmpfs                 19G     0   19G   0% /run/user/67339# packages in environment at /home/ye53nis/.conda/envs/tf-nightly:
    #
    # Name                    Version                   Build  Channel
    _libgcc_mutex             0.1                        main
    absl-py                   0.11.0                   pypi_0    pypi
    alembic                   1.4.1                      py_0    conda-forge/label/main
    appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge/label/main
    argon2-cffi               20.1.0           py38h7b6447c_1
    asn1crypto                1.4.0              pyh9f0ad1d_0    conda-forge/label/main
    asteval                   0.9.16             pyh5ca1d4c_0    conda-forge/label/main
    astunparse                1.6.3                    pypi_0    pypi
    async_generator           1.10               pyhd3eb1b0_0
    attrs                     20.3.0             pyhd3eb1b0_0
    azure-core                1.10.0             pyhd8ed1ab_0    conda-forge/label/main
    azure-storage-blob        12.7.1             pyh44b312d_0    conda-forge/label/main
    backcall                  0.2.0              pyhd3eb1b0_0
    blas                      1.0                         mkl
    bleach                    3.2.3              pyhd3eb1b0_0
    blinker                   1.4                        py_1    conda-forge/label/main
    blosc                     1.20.1               hd408876_0
    brotli                    1.0.9                he6710b0_2
    brotlipy                  0.7.0           py38h27cfd23_1003
    brunsli                   0.1                  h2531618_0
    bzip2                     1.0.8                h7b6447c_0
    ca-certificates           2020.12.5            ha878542_0    conda-forge/label/main
    cachetools                4.2.1                    pypi_0    pypi
    certifi                   2020.12.5        py38h578d9bd_1    conda-forge/label/main
    cffi                      1.14.4           py38h261ae71_0
    chardet                   4.0.0           py38h06a4308_1003
    charls                    2.1.0                he6710b0_2
    click                     7.1.2              pyh9f0ad1d_0    conda-forge/label/main
    cloudpickle               1.6.0                      py_0    conda-forge/label/main
    configparser              5.0.1                      py_0    conda-forge/label/main
    cryptography              3.3.1            py38h3c74f83_0
    cycler                    0.10.0                   py38_0
    databricks-cli            0.9.1                      py_0    conda-forge/label/main
    dbus                      1.13.18              hb2f20db_0
    decorator                 4.4.2              pyhd3eb1b0_0
    defusedxml                0.6.0                      py_0
    docker-py                 4.4.1            py38h578d9bd_1    conda-forge/label/main
    docker-pycreds            0.4.0                      py_0    conda-forge/label/main
    entrypoints               0.3                      py38_0
    expat                     2.2.10               he6710b0_2
    fcsfiles                  2020.9.18                pypi_0    pypi
    flask                     1.1.2              pyh9f0ad1d_0    conda-forge/label/main
    flatbuffers               1.12                     pypi_0    pypi
    fontconfig                2.13.0               h9420a91_0
    freetype                  2.10.4               h5ab3b9f_0
    future                    0.18.2           py38h578d9bd_3    conda-forge/label/main
    gast                      0.4.0                    pypi_0    pypi
    giflib                    5.1.4                h14c3975_1
    gitdb                     4.0.5                      py_0    conda-forge/label/main
    gitpython                 3.1.12             pyhd8ed1ab_0    conda-forge/label/main
    glib                      2.66.1               h92f7085_0
    google-auth               1.24.0                   pypi_0    pypi
    google-auth-oauthlib      0.4.2                    pypi_0    pypi
    google-pasta              0.2.0                    pypi_0    pypi
    gorilla                   0.3.0                      py_0    conda-forge/label/main
    grpcio                    1.34.1                   pypi_0    pypi
    gst-plugins-base          1.14.0               h8213a91_2
    gstreamer                 1.14.0               h28cd5cc_2
    gunicorn                  20.0.4           py38h578d9bd_3    conda-forge/label/main
    h5py                      3.1.0                    pypi_0    pypi
    icu                       58.2                 he6710b0_3
    idna                      2.10               pyhd3eb1b0_0
    imagecodecs               2021.1.11        py38h581e88b_1
    importlib-metadata        2.0.0                      py_1
    importlib_metadata        2.0.0                         1
    intel-openmp              2020.2                      254
    ipykernel                 5.3.4            py38h5ca1d4c_0
    ipython                   7.19.0           py38hb070fc8_1
    ipython_genutils          0.2.0              pyhd3eb1b0_1
    isodate                   0.6.0                      py_1    conda-forge/label/main
    itsdangerous              1.1.0                      py_0    conda-forge/label/main
    jedi                      0.17.0                   py38_0
    jinja2                    2.11.2             pyhd3eb1b0_0
    jpeg                      9b                   h024ee3a_2
    json5                     0.9.5                      py_0
    jsonschema                3.2.0                      py_2
    jupyter_client            6.1.7                      py_0
    jupyter_core              4.7.0            py38h06a4308_0
    jupyterlab                2.2.6                      py_0
    jupyterlab_pygments       0.1.2                      py_0
    jupyterlab_server         1.2.0                      py_0
    jxrlib                    1.1                  h7b6447c_2
    keras-preprocessing       1.1.2                    pypi_0    pypi
    kiwisolver                1.3.0            py38h2531618_0
    lcms2                     2.11                 h396b838_0
    ld_impl_linux-64          2.33.1               h53a641e_7
    lerc                      2.2.1                h2531618_0
    libaec                    1.0.4                he6710b0_1
    libdeflate                1.7                  h27cfd23_5
    libedit                   3.1.20191231         h14c3975_1
    libffi                    3.3                  he6710b0_2
    libgcc-ng                 9.1.0                hdf63c60_0
    libgfortran-ng            7.3.0                hdf63c60_0
    libpng                    1.6.37               hbc83047_0
    libprotobuf               3.13.0.1             h8b12597_0    conda-forge/label/main
    libsodium                 1.0.18               h7b6447c_0
    libstdcxx-ng              9.1.0                hdf63c60_0
    libtiff                   4.1.0                h2733197_1
    libuuid                   1.0.3                h1bed415_2
    libwebp                   1.0.1                h8e7db2f_0
    libxcb                    1.14                 h7b6447c_0
    libxml2                   2.9.10               hb55368b_3
    libzopfli                 1.0.3                he6710b0_0
    lmfit                     1.0.1                      py_1    conda-forge/label/main
    lz4-c                     1.9.3                h2531618_0
    mako                      1.1.4              pyh44b312d_0    conda-forge/label/main
    markdown                  3.3.3                    pypi_0    pypi
    markupsafe                1.1.1            py38h7b6447c_0
    matplotlib                3.3.2                h06a4308_0
    matplotlib-base           3.3.2            py38h817c723_0
    mistune                   0.8.4           py38h7b6447c_1000
    mkl                       2020.2                      256
    mkl-service               2.3.0            py38he904b0f_0
    mkl_fft                   1.2.0            py38h23d657b_0
    mkl_random                1.1.1            py38h0573a6f_0
    mlflow                    1.13.1           py38h578d9bd_2    conda-forge/label/main
    msrest                    0.6.21             pyh44b312d_0    conda-forge/label/main
    multipletau               0.3.3                    pypi_0    pypi
    nbclient                  0.5.1                      py_0
    nbconvert                 6.0.7                    py38_0
    nbformat                  5.1.2              pyhd3eb1b0_1
    ncurses                   6.2                  he6710b0_1
    nest-asyncio              1.4.3              pyhd3eb1b0_0
    notebook                  6.2.0            py38h06a4308_0
    numpy                     1.19.2           py38h54aff64_0
    numpy-base                1.19.2           py38hfa32c7d_0
    oauthlib                  3.0.1                      py_0    conda-forge/label/main
    olefile                   0.46                       py_0
    openjpeg                  2.3.0                h05c96fa_1
    openssl                   1.1.1i               h27cfd23_0
    opt-einsum                3.3.0                    pypi_0    pypi
    packaging                 20.9               pyhd3eb1b0_0
    pandas                    1.2.1            py38ha9443f7_0
    pandoc                    2.11                 hb0f4dca_0
    pandocfilters             1.4.3            py38h06a4308_1
    parso                     0.8.1              pyhd3eb1b0_0
    pcre                      8.44                 he6710b0_0
    pexpect                   4.8.0              pyhd3eb1b0_3
    pickleshare               0.7.5           pyhd3eb1b0_1003
    pillow                    8.1.0            py38he98fc37_0
    pip                       20.3.3           py38h06a4308_0
    prometheus_client         0.9.0              pyhd3eb1b0_0
    prometheus_flask_exporter 0.18.1             pyh9f0ad1d_0    conda-forge/label/main
    prompt-toolkit            3.0.8                      py_0
    protobuf                  3.13.0.1         py38hadf7658_1    conda-forge/label/main
    ptyprocess                0.7.0              pyhd3eb1b0_2
    pyasn1                    0.4.8                    pypi_0    pypi
    pyasn1-modules            0.2.8                    pypi_0    pypi
    pycparser                 2.20                       py_2
    pygments                  2.7.4              pyhd3eb1b0_0
    pyjwt                     2.0.1              pyhd8ed1ab_0    conda-forge/label/main
    pyopenssl                 20.0.1             pyhd3eb1b0_1
    pyparsing                 2.4.7              pyhd3eb1b0_0
    pyqt                      5.9.2            py38h05f1152_4
    pyrsistent                0.17.3           py38h7b6447c_0
    pysocks                   1.7.1            py38h06a4308_0
    python                    3.8.5                h7579374_1
    python-dateutil           2.8.1              pyhd3eb1b0_0
    python-editor             1.0.4                      py_0    conda-forge/label/main
    python_abi                3.8                      1_cp38    conda-forge/label/main
    pytz                      2020.5             pyhd3eb1b0_0
    pyyaml                    5.3.1            py38h8df0ef7_1    conda-forge/label/main
    pyzmq                     20.0.0           py38h2531618_1
    qt                        5.9.7                h5867ecd_1
    querystring_parser        1.2.4                      py_0    conda-forge/label/main
    readline                  8.1                  h27cfd23_0
    requests                  2.25.1             pyhd3eb1b0_0
    requests-oauthlib         1.3.0              pyh9f0ad1d_0    conda-forge/label/main
    rsa                       4.7                      pypi_0    pypi
    scipy                     1.5.2            py38h0b6359f_0
    seaborn                   0.11.1             pyhd3eb1b0_0
    send2trash                1.5.0              pyhd3eb1b0_1
    setuptools                52.0.0           py38h06a4308_0
    sip                       4.19.13          py38he6710b0_0
    six                       1.15.0           py38h06a4308_0
    smmap                     4.0.0              pyh44b312d_0    conda-forge/label/main
    snappy                    1.1.8                he6710b0_0
    sqlalchemy                1.3.20           py38h1e0a361_0    conda-forge/label/main
    sqlite                    3.33.0               h62c20be_0
    sqlparse                  0.4.1              pyh9f0ad1d_0    conda-forge/label/main
    tabulate                  0.8.7              pyh9f0ad1d_0    conda-forge/label/main
    tb-nightly                2.5.0a20210130           pypi_0    pypi
    tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
    termcolor                 1.1.0                    pypi_0    pypi
    terminado                 0.9.2            py38h06a4308_0
    testpath                  0.4.4              pyhd3eb1b0_0
    tf-estimator-nightly      2.5.0.dev2021020101          pypi_0    pypi
    tf-nightly                2.5.0.dev20210130          pypi_0    pypi
    tifffile                  2021.1.14          pyhd3eb1b0_1
    tk                        8.6.10               hbc83047_0
    tornado                   6.1              py38h27cfd23_0
    traitlets                 5.0.5              pyhd3eb1b0_0
    typing-extensions         3.7.4.3                  pypi_0    pypi
    uncertainties             3.1.5              pyhd8ed1ab_0    conda-forge/label/main
    urllib3                   1.26.3             pyhd3eb1b0_0
    wcwidth                   0.2.5                      py_0
    webencodings              0.5.1                    py38_1
    websocket-client          0.57.0           py38h578d9bd_4    conda-forge/label/main
    werkzeug                  1.0.1              pyh9f0ad1d_0    conda-forge/label/main
    wheel                     0.36.2             pyhd3eb1b0_0
    wrapt                     1.12.1                   pypi_0    pypi
    xz                        5.2.5                h7b6447c_0
    yaml                      0.2.5                h516909a_0    conda-forge/label/main
    zeromq                    4.3.3                he6710b0_3
    zfp                       0.5.5                h2531618_4
    zipp                      3.4.0              pyhd3eb1b0_0
    zlib                      1.2.11               h7b6447c_3
    zstd                      1.4.5                h9ceee32_0
    
    Note: you may need to restart the kernel to use updated packages.
    {'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
     'SLURM_NODELIST': 'node260',
     'SLURM_JOB_NAME': 'bash',
     'XDG_SESSION_ID': '595',
     'SLURMD_NODENAME': 'node260',
     'SLURM_TOPOLOGY_ADDR': 'node260',
     'SLURM_NTASKS_PER_NODE': '24',
     'HOSTNAME': 'login01',
     'SLURM_PRIO_PROCESS': '0',
     'SLURM_SRUN_COMM_PORT': '39229',
     'SHELL': '/bin/bash',
     'TERM': 'xterm-color',
     'SLURM_JOB_QOS': 'qstand',
     'SLURM_PTY_WIN_ROW': '39',
     'HISTSIZE': '1000',
     'TMPDIR': '/tmp',
     'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
     'SSH_CLIENT': '10.231.182.213 44428 22',
     'CONDA_SHLVL': '2',
     'CONDA_PROMPT_MODIFIER': '(tf-nightly) ',
     'QTDIR': '/usr/lib64/qt-3.3',
     'QTINC': '/usr/lib64/qt-3.3/include',
     'SSH_TTY': '/dev/pts/2',
     'NO_PROXY': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24',
     'QT_GRAPHICSSYSTEM_CHECKED': '1',
     'SLURM_NNODES': '1',
     'USER': 'ye53nis',
     'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
     'CONDA_EXE': '/cluster/miniconda3/bin/conda',
     'SLURM_STEP_NUM_NODES': '1',
     'SLURM_JOBID': '652176',
     'SRUN_DEBUG': '3',
     'FTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'ftp_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_NTASKS': '24',
     'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
     'SLURM_STEP_ID': '0',
     'TMUX': '/tmp/tmux-67339/default,43792,2',
     '_CE_CONDA': '',
     'CONDA_PREFIX_1': '/cluster/miniconda3',
     'SLURM_STEP_LAUNCHER_PORT': '39229','SLURM_TASKS_PER_NODE': '24',
    'MAIL': '/var/spool/mail/ye53nis',
    'PATH': '/home/ye53nis/.conda/envs/tf-nightly/bin:/home/lex/Programme/miniconda3/envs/tf-nightly-lab/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
    'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
    'SLURM_JOB_ID': '652176',
    'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tf-nightly',
    'SLURM_JOB_USER': 'ye53nis',
    'SLURM_STEPID': '0',
    'PWD': '/',
    'SLURM_SRUN_COMM_HOST': '192.168.192.5',
    'LANG': 'en_US.UTF-8',
    'SLURM_PTY_WIN_COL': '205',
    'SLURM_UMASK': '0022',
    'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
    'SLURM_JOB_UID': '67339',
    'LOADEDMODULES': '',
    'SLURM_NODEID': '0',
    'TMUX_PANE': '%2',
    'SLURM_SUBMIT_DIR': '/',
    'SLURM_TASK_PID': '167340',
    'SLURM_NPROCS': '24',
    'SLURM_CPUS_ON_NODE': '24',
    'SLURM_DISTRIBUTION': 'block',
    'HTTPS_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
    'https_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
    'SLURM_PROCID': '0',
    'HISTCONTROL': 'ignoredups',
    '_CE_M': '',
    'SLURM_JOB_NODELIST': 'node260',
    'SLURM_PTY_PORT': '33620',
    'HOME': '/home/ye53nis',
    'SHLVL': '3',
    'SLURM_LOCALID': '0',
    'SLURM_JOB_GID': '13280',
    'SLURM_JOB_CPUS_PER_NODE': '24',
    'SLURM_CLUSTER_NAME': 'hpc',
    'no_proxy': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24',
    'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
    'SLURM_SUBMIT_HOST': 'login01',
    'HTTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
    'SLURM_JOB_PARTITION': 's_standard',
    'MATHEMATICA_HOME': '/cluster/apps/mathematica/11.3',
    'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
    'LOGNAME': 'ye53nis',
    'SLURM_STEP_NUM_TASKS': '24',
    'QTLIB': '/usr/lib64/qt-3.3/lib',
    'SLURM_JOB_ACCOUNT': 'iaob',
    'SLURM_JOB_NUM_NODES': '1',
    'MODULESHOME': '/usr/share/Modules',
    'CONDA_DEFAULT_ENV': 'tf-nightly',
    'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
    'SLURM_STEP_TASKS_PER_NODE': '24',
    'PORT': '9999',
    'SLURM_STEP_NODELIST': 'node260',
    'DISPLAY': ':0',
    'XDG_RUNTIME_DIR': '',
    'XAUTHORITY': '/home/lex/.Xauthority',
    'BASH_FUNC_module()': '() {  eval `/usr/bin/modulecmd bash $*`\n}',
    '_': '/home/ye53nis/.conda/envs/tf-nightly/bin/jupyter',
    'JPY_PARENT_PID': '446797',
    'CLICOLOR': '1',
    'PAGER': 'cat',
    'GIT_PAGER': 'cat',
    'MPLBACKEND': 'module://ipykernel.pylab.backend_inline'}
    
2.3.5.2 Set mlflow variables
  • mlflow environment variables
    conda activate tf-nightly
    cd /beegfs/ye53nis/drmed-git
    export MLFLOW_EXPERIMENT_NAME=exp-210204-unet
    export MLFLOW_TRACKING_URI=file:./data/mlruns
    mkdir data/exp-210204-unet
    
    (tf-nightly) [ye53nis@node117 drmed-git]$
    
2.3.5.3 run mlflow
  • Use whole dataset (6400 training, 1600 validation, 2000 test), but during training, use only 1/5th of it per epoch - but this time with more balanced test set.
    mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=50 -P learning_rate=None -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train -P csv_path_test=/beegfs/ye53nis/saves/firstartifact_Nov2020_test -P steps_per_epoch=1280 -P validation_steps=320
    
    (tf-nightly) [ye53nis@node117 drmed-git]$ mlflow run . -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P epochs=50 -P learning_rate=None -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train -P csv_path_test=/beegfs/y
    e53nis/saves/firstartifact_Nov2020_test -P steps_per_epoch=1280 -P validation_steps=320
    WARNING:root:Malformed experiment '1'. Detailed error Yaml file './data/mlruns/1/meta.yaml' does not exist.
    Traceback (most recent call last):
      File "/home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 237, in list_experiments
        experiment = self._get_experiment(exp_id, view_type)
      File "/home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 311, in _get_experiment
        meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
      File "/home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 170, in read_yaml
        raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
    mlflow.exceptions.MissingConfigException: Yaml file './data/mlruns/1/meta.yaml' does not exist.
    WARNING:root:Malformed experiment '0'. Detailed error Yaml file './data/mlruns/0/meta.yaml' does not exist.
    Traceback (most recent call last):
      File "/home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 237, in list_experiments
        experiment = self._get_experiment(exp_id, view_type)
      File "/home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 311, in _get_experiment
        meta = read_yaml(experiment_dir, FileStore.META_DATA_FILE_NAME)
      File "/home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/mlflow/utils/file_utils.py", line 170, in read_yaml
        raise MissingConfigException("Yaml file '%s' does not exist." % file_path)
    mlflow.exceptions.MissingConfigException: Yaml file './data/mlruns/0/meta.yaml' does not exist.
    2021/03/16 22:26:14 INFO mlflow.projects.utils: === Created directory /tmp/tmp4e89kbom for downloading remote URIs passed to arguments of type 'path' ===
    2021/03/16 22:26:14 INFO mlflow.projects.backend.local: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-fd9a200e5a24d4c79a0ff13be73ccb5141ed072c 1>&2 && python src/fluotracify/trai
    ning/train.py /beegfs/ye53nis/drmed-git/src 5 0.2 16384 None 50 /beegfs/ye53nis/saves/firstartifact_Nov2020_train /beegfs/ye53nis/saves/firstartifact_Nov2020_test 3 1280 320' in run with ID '3cec3f26ed2d4004978c4ec37c00fba0' ===
    2021-03-16 22:26:31.835407: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2021-03-16 22:26:31.835484: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    2.5.0-dev20210130
    2021-03-16 22:27:03.495143: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2021-03-16 22:27:03.495217: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
    2021-03-16 22:27:03.495359: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node117): /proc/driver/nvidia/version does not exist
    GPUs:  []
    train 0 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/50/traces_brightclust_Nov2020_D50_set005.csv
    train 1 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.6/traces_brightclust_Nov2020_D0.6_set001.csv
    train 2 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/3.0/traces_brightclust_Nov2020_D3.0_set005.csv
    train 3 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/50/traces_brightclust_Nov2020_D50_set008.csv
    train 4 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/10/traces_brightclust_Nov2020_D10_set004.csv
    train 5 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/1.0/traces_brightclust_Nov2020_D1.0_set004.csv
    train 6 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.08/traces_brightclust_Nov2020_D0.08_set004.csv
    train 7 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/50/traces_brightclust_Nov2020_D50_set004.csv
    train 8 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/50/traces_brightclust_Nov2020_D50_set009.csv
    train 9 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
    train 10 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.4/traces_brightclust_Nov2020_D0.4_set002.csv
    train 11 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.4/traces_brightclust_Nov2020_D0.4_set006.csv
    train 12 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.4/traces_brightclust_Nov2020_D0.4_set010.csv
    train 13 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.4/traces_brightclust_Nov2020_D0.4_set007.csv
    train 14 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.2/traces_brightclust_Nov2020_D0.2_set004.csv
    train 15 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/10/traces_brightclust_Nov2020_D10_set008.csv
    train 16 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/3.0/traces_brightclust_Nov2020_D3.0_set009.csv
    train 17 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
    train 18 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.069/traces_brightclust_Nov2020_D0.069_set008.csv
    train 19 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.2/traces_brightclust_Nov2020_D0.2_set010.csv
    train 20 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.4/traces_brightclust_Nov2020_D0.4_set003.csv
    train 21 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.069/traces_brightclust_Nov2020_D0.069_set006.csv
    train 22 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/10/traces_brightclust_Nov2020_D10_set010.csv
    train 23 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.08/traces_brightclust_Nov2020_D0.08_set009.csv
    train 24 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.069/traces_brightclust_Nov2020_D0.069_set007.csv
    train 25 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/10/traces_brightclust_Nov2020_D10_set009.csv
    train 26 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/50/traces_brightclust_Nov2020_D50_set010.csv
    train 27 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.08/traces_brightclust_Nov2020_D0.08_set008.csv
    train 28 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.6/traces_brightclust_Nov2020_D0.6_set010.csv
    train 29 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/10/traces_brightclust_Nov2020_D10_set003.csv
    train 30 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
    train 31 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/3.0/traces_brightclust_Nov2020_D3.0_set003.csv
    train 32 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.4/traces_brightclust_Nov2020_D0.4_set009.csv
    train 33 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.1/traces_brightclust_Nov2020_D0.1_set004.csv
    train 34 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/3.0/traces_brightclust_Nov2020_D3.0_set001.csv
    train 35 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.1/traces_brightclust_Nov2020_D0.1_set010.csv
    train 36 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.4/traces_brightclust_Nov2020_D0.4_set004.csv
    train 37 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/10/traces_brightclust_Nov2020_D10_set007.csv
    train 38 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.6/traces_brightclust_Nov2020_D0.6_set002.csv
    train 39 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.1/traces_brightclust_Nov2020_D0.1_set009.csv
    train 40 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.069/traces_brightclust_Nov2020_D0.069_set002.csv
    train 41 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.1/traces_brightclust_Nov2020_D0.1_set006.csv
    train 42 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.069/traces_brightclust_Nov2020_D0.069_set009.csv
    train 43 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.1/traces_brightclust_Nov2020_D0.1_set007.csv
    train 44 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.2/traces_brightclust_Nov2020_D0.2_set001.csv
    train 45 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/50/traces_brightclust_Nov2020_D50_set007.csv
    train 46 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.08/traces_brightclust_Nov2020_D0.08_set006.csv
    train 47 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.1/traces_brightclust_Nov2020_D0.1_set003.csv
    train 48 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.2/traces_brightclust_Nov2020_D0.2_set009.csv
    train 49 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.6/traces_brightclust_Nov2020_D0.6_set005.csv
    train 50 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.1/traces_brightclust_Nov2020_D0.1_set008.csv
    train 51 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/3.0/traces_brightclust_Nov2020_D3.0_set006.csv
    train 52 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/1.0/traces_brightclust_Nov2020_D1.0_set009.csv
    train 53 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/3.0/traces_brightclust_Nov2020_D3.0_set010.csv
    train 54 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.6/traces_brightclust_Nov2020_D0.6_set007.csv
    train 55 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.6/traces_brightclust_Nov2020_D0.6_set006.csv
    train 56 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.069/traces_brightclust_Nov2020_D0.069_set003.csv
    train 57 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.08/traces_brightclust_Nov2020_D0.08_set010.csv
    train 58 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/3.0/traces_brightclust_Nov2020_D3.0_set008.csv
    train 59 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.2/traces_brightclust_Nov2020_D0.2_set008.csv
    train 60 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.08/traces_brightclust_Nov2020_D0.08_set002.csv
    train 61 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.2/traces_brightclust_Nov2020_D0.2_set006.csv
    train 62 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.6/traces_brightclust_Nov2020_D0.6_set004.csv
    train 63 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/10/traces_brightclust_Nov2020_D10_set006.csv
    train 64 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.2/traces_brightclust_Nov2020_D0.2_set003.csv
    train 65 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/0.08/traces_brightclust_Nov2020_D0.08_set007.csv
    train 66 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/50/traces_brightclust_Nov2020_D50_set006.csv
    train 67 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/1.0/traces_brightclust_Nov2020_D1.0_set010.csv
    train 68 /beegfs/ye53nis/saves/firstartifact_Nov2020_train/1.0/traces_brightclust_Nov2020_D1.0_set007.csv
    train 0 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.069_set010.csv
    train 1 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D50_set002.csv
    train 2 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.4_set005.csv
    train 3 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.2_set005.csv
    train 4 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D3.0_set007.csv
    train 5 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D3.0_set002.csv
    train 6 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D50_set001.csv
    train 7 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.2_set007.csv
    train 8 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.6_set009.csv
    train 9 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D10_set002.csv
    train 10 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.08_set005.csv
    train 11 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.6_set008.csv
    train 12 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.1_set005.csv
    train 13 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.4_set008.csv
    train 14 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D10_set005.csv
    train 15 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D1.0_set006.csv
    train 16 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.069_set005.csv
    train 17 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D50_set003.csv
    train 18 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.1_set001.csv
    train 19 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.08_set003.csv
    train 20 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D1.0_set003.csv
    train 21 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D1.0_set005.csv
    train 22 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.2_set002.csv
    train 23 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.1_set002.csv
    train 24 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D3.0_set004.csv
    train 25 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.08_set001.csv
    train 26 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.069_set001.csv
    train 27 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D10_set001.csv
    train 28 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.6_set003.csv
    train 29 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.4_set001.csv
    The given DataFrame was split into 3 parts with shapes: [(16384, 6900), (16384, 6900), (16384, 6900)]
    The given DataFrame was split into 3 parts with shapes: [(16384, 3000), (16384, 3000), (16384, 3000)]
    
    for each 16384 timestap trace there are the following numbers of corrupted timesteps:
    label001_1    1916
    label002_1    1004
    label003_1    1476
    label004_1    1154
    label005_1    1454
    dtype: int64
    2021-03-16 22:34:23.544689: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operati
    ons:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    number of training examples: 5520, number of validation examples: 1380
    
    ------------------------
    number of test examples: 3000
    
    input - shape:   (None, 16384, 1)
    output - shape:  (None, 16384, 1)
    2021-03-16 22:34:30.005739: I tensorflow/core/profiler/lib/profiler_session.cc:126] Profiler session initializing.
    2021-03-16 22:34:30.005803: I tensorflow/core/profiler/lib/profiler_session.cc:141] Profiler session started.
    2021-03-16 22:34:30.005880: I tensorflow/core/profiler/lib/profiler_session.cc:158] Profiler session tear down.
    2021/03/16 22:34:30 INFO mlflow.utils.autologging_utils: tensorflow autologging will track hyperparameters, performance metrics, model artifacts, and lineage information for the current tensorflow workflow to the MLflow run with ID '3c
    ec3f26ed2d4004978c4ec37c00fba0'
    2021/03/16 22:34:30 WARNING mlflow.utils.autologging_utils: MLflow issued a warning during tensorflow autologging: "/home/ye53nis/.conda/envs/mlflow-fd9a200e5a24d4c79a0ff13be73ccb5141ed072c/lib/python3.8/site-packages/mlflow/utils/auto
    logging_utils.py:86: UserWarning: Logging to MLflow failed: Changing param values is not allowed. Param with key='batch_size' was already logged with value='5' for run ID='3cec3f26ed2d4004978c4ec37c00fba0'. Attempted logging new value
    'None'."
    Epoch 1/50
    2021-03-16 22:34:43.027454: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:145] None of the MLIR Optimization Passes are enabled (registered 2)
    2021-03-16 22:34:43.259103: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2194880000 Hz
       1/1280 [..............................] - ETA: 7:06:43 - loss: 1.7012 - tp0.1: 11443.0000 - fp0.1: 69168.0000 - tn0.1: 50.0000 - fn0.1: 1259.0000 - precision0.1: 0.1420 - recall0.1: 0.9009 - tp0.3: 9371.0000 - fp0.3: 67996.0000 - tn
    0.3: 1222.0000 - fn0.3: 3331.0000 - precision0.3: 0.1211 - recall0.3: 0.7378 - tp0.5: 5565.0000 - fp0.5: 37117.0000 - tn0.5: 32101.0000 - fn0.5: 7137.0000 - precision0.5: 0.1304 - recall0.5: 0.4381 - tp0.7: 1504.0000 - fp0.7: 2293.0000
     - tn0.7: 66925.0000 - fn0.7: 11198.0000 - precision0.7: 0.3961 - recall0.7: 0.1184 - tp0.9: 284.0000 - fp0.9: 114.0000 - tn0.9: 69104.0000 - fn0.9: 12418.0000 - precision0.9: 0.7136 - recall0.9: 0.0224 - accuracy: 0.4598 - auc: 0.4176
    2021-03-16 22:34:50.791836: I tensorflow/core/profiler/lib/profiler_session.cc:126] Profiler session initializing.
    2021-03-16 22:34:50.791935: I tensorflow/core/profiler/lib/profiler_session.cc:141] Profiler session started.
       2/1280 [..............................] - ETA: 55:09 - loss: 1.5706 - tp0.1: 15933.5000 - fp0.1: 91410.0000 - tn0.1: 14176.5000 - fn0.1: 1360.0000 - precision0.1: 0.1471 - recall0.1: 0.9171 - tp0.3: 11355.0000 - fp0.3: 68009.0000 -
    tn0.3: 37577.5000 - fn0.3: 5938.5000 - precision0.3: 0.1425 - recall0.3: 0.6736 - tp0.5: 7264.0000 - fp0.5: 37126.5000 - tn0.5: 68460.0000 - fn0.5: 10029.5000 - precision0.5: 0.1624 - recall0.5: 0.4238 - tp0.7: 2845.0000 - fp0.7: 2300.
    0000 - tn0.7: 103286.5000 - fn0.7: 14448.5000 - precision0.7: 0.5204 - recall0.7: 0.1548 - tp0.9: 1306.5000 - fp0.9: 119.0000 - tn0.9: 105467.5000 - fn0.9: 15987.0000 - precision0.9: 0.8315 - recall0.9: 0.0644 - accuracy: 0.5771 - auc:
     0.53722021-03-16 22:34:53.384001: I tensorflow/core/profiler/lib/profiler_session.cc:66] Profiler session collecting data.
    2021-03-16 22:34:53.414299: I tensorflow/core/profiler/lib/profiler_session.cc:158] Profiler session tear down.
    2021-03-16 22:34:53.449110: I tensorflow/core/profiler/rpc/client/save_profile.cc:137] Creating directory: /tmp/tb/train/plugins/profile/2021_03_16_22_34_53
    2021-03-16 22:34:53.468888: I tensorflow/core/profiler/rpc/client/save_profile.cc:143] Dumped gzipped tool data for trace.json.gz to /tmp/tb/train/plugins/profile/2021_03_16_22_34_53/node117.trace.json.gz
    2021-03-16 22:34:53.498618: I tensorflow/core/profiler/rpc/client/save_profile.cc:137] Creating directory: /tmp/tb/train/plugins/profile/2021_03_16_22_34_53
    2021-03-16 22:34:53.498775: I tensorflow/core/profiler/rpc/client/save_profile.cc:143] Dumped gzipped tool data for memory_profile.json.gz to /tmp/tb/train/plugins/profile/2021_03_16_22_34_53/node117.memory_profile.json.gz
    2021-03-16 22:34:53.501218: I tensorflow/core/profiler/rpc/client/capture_profile.cc:251] Creating directory: /tmp/tb/train/plugins/profile/2021_03_16_22_34_53Dumped tool data for xplane.pb to /tmp/tb/train/plugins/profile/2021_03_16_2
    2_34_53/node117.xplane.pb
    Dumped tool data for overview_page.pb to /tmp/tb/train/plugins/profile/2021_03_16_22_34_53/node117.overview_page.pb
    Dumped tool data for input_pipeline.pb to /tmp/tb/train/plugins/profile/2021_03_16_22_34_53/node117.input_pipeline.pb
    Dumped tool data for tensorflow_stats.pb to /tmp/tb/train/plugins/profile/2021_03_16_22_34_53/node117.tensorflow_stats.pb
    Dumped tool data for kernel_stats.pb to /tmp/tb/train/plugins/profile/2021_03_16_22_34_53/node117.kernel_stats.pb
    
    1280/1280 [==============================] - 3356s 3s/step - loss: 0.9438 - tp0.1: 7090847.5004 - fp0.1: 12587220.8876 - tn0.1: 31719449.0960 - fn0.1: 1113141.3841 - precision0.1: 0.3522 - recall0.1: 0.8539 - tp0.3: 5944647.5215 - fp0.
    3: 7704502.4426 - tn0.3: 36602146.0117 - fn0.3: 2259341.3630 - precision0.3: 0.4254 - recall0.3: 0.6980 - tp0.5: 3390349.7166 - fp0.5: 2172177.0687 - tn0.5: 42134502.0273 - fn0.5: 4813639.1678 - precision0.5: 0.5845 - recall0.5: 0.4054
     - tp0.7: 2075799.0164 - fp0.7: 570841.5262 - tn0.7: 43735856.3763 - fn0.7: 6128189.8681 - precision0.7: 0.7474 - recall0.7: 0.2354 - tp0.9: 1317276.8236 - fp0.9: 157422.0055 - tn0.9: 44149241.1343 - fn0.9: 6886712.0609 - precision0.9:
     0.8734 - recall0.9: 0.1441 - accuracy: 0.8611 - auc: 0.8419 - val_loss: 1.7438 - val_tp0.1: 1858818.0000 - val_fp0.1: 1751074.0000 - val_tn0.1: 20297196.0000 - val_fn0.1: 2307317.0000 - val_precision0.1: 0.5149 - val_recall0.1: 0.4462
     - val_tp0.3: 1147351.0000 - val_fp0.3: 857162.0000 - val_tn0.3: 21191096.0000 - val_fn0.3: 3018784.0000 - val_precision0.3: 0.5724 - val_recall0.3: 0.2754 - val_tp0.5: 22832.0000 - val_fp0.5: 2666.0000 - val_tn0.5: 22045600.0000 - val
    _fn0.5: 4143303.0000 - val_precision0.5: 0.8954 - val_recall0.5: 0.0055 - val_tp0.7: 0.0000e+00 - val_fp0.7: 0.0000e+00 - val_tn0.7: 22048264.0000 - val_fn0.7: 4166135.0000 - val_precision0.7: 0.0000e+00 - val_recall0.7: 0.0000e+00 - v
    al_tp0.9: 0.0000e+00 - val_fp0.9: 0.0000e+00 - val_tn0.9: 22048264.0000 - val_fn0.9: 4166135.0000 - val_precision0.9: 0.0000e+00 - val_recall0.9: 0.0000e+00 - val_accuracy: 0.8418 - val_auc: 0.7129
    Epoch 2/50
    1280/1280 [==============================] - 3273s 3s/step - loss: 0.7454 - tp0.1: 7319058.9383 - fp0.1: 12182101.0671 - tn0.1: 32209636.0234 - fn0.1: 799871.6300 - precision0.1: 0.3732 - recall0.1: 0.9003 - tp0.3: 6349950.7744 - fp0.3
    : 7864965.2303 - tn0.3: 36526747.8587 - fn0.3: 1768979.7939 - precision0.3: 0.4426 - recall0.3: 0.7794 - tp0.5: 3304850.4598 - fp0.5: 1018320.5308 - tn0.5: 43373388.4637 - fn0.5: 4814080.1085 - precision0.5: 0.7573 - recall0.5: 0.4070
    - tp0.7: 2713284.5909 - fp0.7: 370786.4668 - tn0.7: 44020933.5667 - fn0.7: 5405645.9774 - precision0.7: 0.8821 - recall0.7: 0.3298 - tp0.9: 2048963.3240 - fp0.9: 143660.5839 - tn0.9: 44248083.9859 - fn0.9: 6069967.2443 - precision0.9:
    0.9325 - recall0.9: 0.2463 - accuracy: 0.8888 - auc: 0.8781 - val_loss: 1.8629 - val_tp0.1: 1696755.0000 - val_fp0.1: 1369278.0000 - val_tn0.1: 20627520.0000 - val_fn0.1: 2520850.0000 - val_precision0.1: 0.5534 - val_recall0.1: 0.4023
    - val_tp0.3: 1177823.0000 - val_fp0.3: 724329.0000 - val_tn0.3: 21272468.0000 - val_fn0.3: 3039782.0000 - val_precision0.3: 0.6192 - val_recall0.3: 0.2793 - val_tp0.5: 726105.0000 - val_fp0.5: 220632.0000 - val_tn0.5: 21776148.0000 - v
    al_fn0.5: 3491500.0000 - val_precision0.5: 0.7670 - val_recall0.5: 0.1722 - val_tp0.7: 622624.0000 - val_fp0.7: 149383.0000 - val_tn0.7: 21847414.0000 - val_fn0.7: 3594981.0000 - val_precision0.7: 0.8065 - val_recall0.7: 0.1476 - val_t
    p0.9: 478068.0000 - val_fp0.9: 78711.0000 - val_tn0.9: 21918088.0000 - val_fn0.9: 3739537.0000 - val_precision0.9: 0.8586 - val_recall0.9: 0.1134 - val_accuracy: 0.8584 - val_auc: 0.6881
    Epoch 3/50
    1280/1280 [==============================] - 3268s 3s/step - loss: 0.6401 - tp0.1: 7574454.9469 - fp0.1: 10689439.2311 - tn0.1: 33573907.5972 - fn0.1: 672847.0898 - precision0.1: 0.4156 - recall0.1: 0.9171 - tp0.3: 6542181.5066 - fp0.3
    : 5029903.7557 - tn0.3: 39233452.5995 - fn0.3: 1705120.5301 - precision0.3: 0.5571 - recall0.3: 0.7912 - tp0.5: 4316700.4192 - fp0.5: 813585.7369 - tn0.5: 43449769.7229 - fn0.5: 3930601.6175 - precision0.5: 0.8494 - recall0.5: 0.5079 -
     tp0.7: 3862233.1499 - fp0.7: 434886.9867 - tn0.7: 43828435.6581 - fn0.7: 4385068.8868 - precision0.7: 0.9012 - recall0.7: 0.4570 - tp0.9: 3000635.4949 - fp0.9: 146691.9102 - tn0.9: 44116668.4348 - fn0.9: 5246666.5418 - precision0.9: 0
    .9538 - recall0.9: 0.3597 - accuracy: 0.9073 - auc: 0.9113 - val_loss: 1.2681 - val_tp0.1: 2112447.0000 - val_fp0.1: 1372411.0000 - val_tn0.1: 20688214.0000 - val_fn0.1: 2041318.0000 - val_precision0.1: 0.6062 - val_recall0.1: 0.5086 -
     val_tp0.3: 1459147.0000 - val_fp0.3: 526021.0000 - val_tn0.3: 21534608.0000 - val_fn0.3: 2694618.0000 - val_precision0.3: 0.7350 - val_recall0.3: 0.3513 - val_tp0.5: 1206684.0000 - val_fp0.5: 343698.0000 - val_tn0.5: 21716924.0000 - v
    al_fn0.5: 2947081.0000 - val_precision0.5: 0.7783 - val_recall0.5: 0.2905 - val_tp0.7: 926901.0000 - val_fp0.7: 212441.0000 - val_tn0.7: 21848180.0000 - val_fn0.7: 3226864.0000 - val_precision0.7: 0.8135 - val_recall0.7: 0.2231 - val_t
    p0.9: 701556.0000 - val_fp0.9: 119958.0000 - val_tn0.9: 21940680.0000 - val_fn0.9: 3452209.0000 - val_precision0.9: 0.8540 - val_recall0.9: 0.1689 - val_accuracy: 0.8745 - val_auc: 0.7587
    Epoch 4/50
    1280/1280 [==============================] - 3266s 3s/step - loss: 0.5483 - tp0.1: 7577427.0859 - fp0.1: 8351943.7939 - tn0.1: 35940757.2381 - fn0.1: 640533.9243 - precision0.1: 0.4704 - recall0.1: 0.9241 - tp0.3: 6848424.7369 - fp0.3:
     4027475.1538 - tn0.3: 40265212.3185 - fn0.3: 1369536.2732 - precision0.3: 0.6241 - recall0.3: 0.8352 - tp0.5: 6060996.3591 - fp0.5: 2313256.1897 - tn0.5: 41979441.1702 - fn0.5: 2156964.6511 - precision0.5: 0.7177 - recall0.5: 0.7389 -
     tp0.7: 4556251.4965 - fp0.7: 542255.6714 - tn0.7: 43750437.9961 - fn0.7: 3661709.5137 - precision0.7: 0.8912 - recall0.7: 0.5511 - tp0.9: 3395492.4715 - fp0.9: 154251.2139 - tn0.9: 44138439.1054 - fn0.9: 4822468.5386 - precision0.9: 0
    .9548 - recall0.9: 0.4106 - accuracy: 0.9134 - auc: 0.9349 - val_loss: 1.5831 - val_tp0.1: 3220872.0000 - val_fp0.1: 3814830.0000 - val_tn0.1: 18497596.0000 - val_fn0.1: 681105.0000 - val_precision0.1: 0.4578 - val_recall0.1: 0.8254 -
    val_tp0.3: 2365990.0000 - val_fp0.3: 2561983.0000 - val_tn0.3: 19750440.0000 - val_fn0.3: 1535987.0000 - val_precision0.3: 0.4801 - val_recall0.3: 0.6064 - val_tp0.5: 1952532.0000 - val_fp0.5: 2291714.0000 - val_tn0.5: 20020708.0000 -
    val_fn0.5: 1949445.0000 - val_precision0.5: 0.4600 - val_recall0.5: 0.5004 - val_tp0.7: 1618270.0000 - val_fp0.7: 2113505.0000 - val_tn0.7: 20198920.0000 - val_fn0.7: 2283707.0000 - val_precision0.7: 0.4336 - val_recall0.7: 0.4147 - va
    l_tp0.9: 1467404.0000 - val_fp0.9: 1996280.0000 - val_tn0.9: 20316148.0000 - val_fn0.9: 2434573.0000 - val_precision0.9: 0.4237 - val_recall0.9: 0.3761 - val_accuracy: 0.8382 - val_auc: 0.8531
    Epoch 5/50
    1280/1280 [==============================] - 3259s 3s/step - loss: 0.4575 - tp0.1: 7563591.0968 - fp0.1: 6166461.2155 - tn0.1: 38207695.0180 - fn0.1: 572899.0141 - precision0.1: 0.5449 - recall0.1: 0.9269 - tp0.3: 6976500.9703 - fp0.3:
     2853572.8470 - tn0.3: 41520584.7908 - fn0.3: 1159989.1405 - precision0.3: 0.7064 - recall0.3: 0.8492 - tp0.5: 6358931.4910 - fp0.5: 1545188.1085 - tn0.5: 42828968.9243 - fn0.5: 1777558.6198 - precision0.5: 0.8010 - recall0.5: 0.7718 -
     tp0.7: 5171541.8845 - fp0.7: 510104.5824 - tn0.7: 43864075.8119 - fn0.7: 2964948.2264 - precision0.7: 0.9101 - recall0.7: 0.6254 - tp0.9: 3989419.1991 - fp0.9: 146867.6534 - tn0.9: 44227286.8977 - fn0.9: 4147070.9118 - precision0.9: 0
    .9658 - recall0.9: 0.4744 - accuracy: 0.9360 - auc: 0.9496 - val_loss: 1.0113 - val_tp0.1: 2745883.0000 - val_fp0.1: 1806365.0000 - val_tn0.1: 20277030.0000 - val_fn0.1: 1385129.0000 - val_precision0.1: 0.6032 - val_recall0.1: 0.6647 -
     val_tp0.3: 1898747.0000 - val_fp0.3: 1004741.0000 - val_tn0.3: 21078656.0000 - val_fn0.3: 2232265.0000 - val_precision0.3: 0.6540 - val_recall0.3: 0.4596 - val_tp0.5: 1378572.0000 - val_fp0.5: 684339.0000 - val_tn0.5: 21399046.0000 -
    val_fn0.5: 2752440.0000 - val_precision0.5: 0.6683 - val_recall0.5: 0.3337 - val_tp0.7: 1058802.0000 - val_fp0.7: 422419.0000 - val_tn0.7: 21660976.0000 - val_fn0.7: 3072210.0000 - val_precision0.7: 0.7148 - val_recall0.7: 0.2563 - val
    _tp0.9: 633251.0000 - val_fp0.9: 145724.0000 - val_tn0.9: 21937676.0000 - val_fn0.9: 3497761.0000 - val_precision0.9: 0.8129 - val_recall0.9: 0.1533 - val_accuracy: 0.8689 - val_auc: 0.8229
    Epoch 6/50
    1280/1280 [==============================] - 3293s 3s/step - loss: 0.3999 - tp0.1: 7670968.6714 - fp0.1: 5321303.1358 - tn0.1: 38997617.0039 - fn0.1: 520751.9563 - precision0.1: 0.5842 - recall0.1: 0.9342 - tp0.3: 7188663.1733 - fp0.3:
     2542544.6495 - tn0.3: 41776414.2037 - fn0.3: 1003057.4543 - precision0.3: 0.7341 - recall0.3: 0.8729 - tp0.5: 6622211.5550 - fp0.5: 1333118.7088 - tn0.5: 42985778.1749 - fn0.5: 1569509.0726 - precision0.5: 0.8305 - recall0.5: 0.8017 -
     tp0.7: 5590775.7127 - fp0.7: 478800.2568 - tn0.7: 43840116.4879 - fn0.7: 2600944.9149 - precision0.7: 0.9198 - recall0.7: 0.6746 - tp0.9: 4639602.3575 - fp0.9: 154184.4598 - tn0.9: 44164751.2225 - fn0.9: 3552118.2701 - precision0.9: 0
    .9678 - recall0.9: 0.5580 - accuracy: 0.9436 - auc: 0.9557 - val_loss: 0.7256 - val_tp0.1: 3301232.0000 - val_fp0.1: 2777637.0000 - val_tn0.1: 19396900.0000 - val_fn0.1: 738630.0000 - val_precision0.1: 0.5431 - val_recall0.1: 0.8172 -
    val_tp0.3: 2789396.0000 - val_fp0.3: 1963405.0000 - val_tn0.3: 20211132.0000 - val_fn0.3: 1250466.0000 - val_precision0.3: 0.5869 - val_recall0.3: 0.6905 - val_tp0.5: 2332065.0000 - val_fp0.5: 1478069.0000 - val_tn0.5: 20696466.0000 -
    val_fn0.5: 1707797.0000 - val_precision0.5: 0.6121 - val_recall0.5: 0.5773 - val_tp0.7: 1949962.0000 - val_fp0.7: 1076978.0000 - val_tn0.7: 21097560.0000 - val_fn0.7: 2089900.0000 - val_precision0.7: 0.6442 - val_recall0.7: 0.4827 - va
    l_tp0.9: 1380432.0000 - val_fp0.9: 492826.0000 - val_tn0.9: 21681716.0000 - val_fn0.9: 2659430.0000 - val_precision0.9: 0.7369 - val_recall0.9: 0.3417 - val_accuracy: 0.8785 - val_auc: 0.8740
    Epoch 7/50
    1280/1280 [==============================] - 3286s 3s/step - loss: 0.3597 - tp0.1: 7890232.5785 - fp0.1: 4953542.9781 - tn0.1: 39218777.5909 - fn0.1: 448103.7330 - precision0.1: 0.6178 - recall0.1: 0.9465 - tp0.3: 7467308.2701 - fp0.3:
     2391275.9922 - tn0.3: 41781019.9024 - fn0.3: 871028.0414 - precision0.3: 0.7613 - recall0.3: 0.8946 - tp0.5: 6988139.4286 - fp0.5: 1302055.8244 - tn0.5: 42870220.7486 - fn0.5: 1350196.8829 - precision0.5: 0.8451 - recall0.5: 0.8377 -
    tp0.7: 5947552.7845 - fp0.7: 472146.9633 - tn0.7: 43700160.4223 - fn0.7: 2390783.5269 - precision0.7: 0.9259 - recall0.7: 0.7138 - tp0.9: 4968978.2139 - fp0.9: 143981.8579 - tn0.9: 44028364.7588 - fn0.9: 3369358.0976 - precision0.9: 0.
    9713 - recall0.9: 0.5953 - accuracy: 0.9493 - auc: 0.9636 - val_loss: 0.7770 - val_tp0.1: 3234726.0000 - val_fp0.1: 2249439.0000 - val_tn0.1: 19829152.0000 - val_fn0.1: 901082.0000 - val_precision0.1: 0.5898 - val_recall0.1: 0.7821 - v
    al_tp0.3: 2554897.0000 - val_fp0.3: 1502825.0000 - val_tn0.3: 20575758.0000 - val_fn0.3: 1580911.0000 - val_precision0.3: 0.6296 - val_recall0.3: 0.6178 - val_tp0.5: 1992390.0000 - val_fp0.5: 1136809.0000 - val_tn0.5: 20941788.0000 - v
    al_fn0.5: 2143418.0000 - val_precision0.5: 0.6367 - val_recall0.5: 0.4817 - val_tp0.7: 1526404.0000 - val_fp0.7: 689435.0000 - val_tn0.7: 21389156.0000 - val_fn0.7: 2609404.0000 - val_precision0.7: 0.6889 - val_recall0.7: 0.3691 - val_
    tp0.9: 847679.0000 - val_fp0.9: 133403.0000 - val_tn0.9: 21945180.0000 - val_fn0.9: 3288129.0000 - val_precision0.9: 0.8640 - val_recall0.9: 0.2050 - val_accuracy: 0.8749 - val_auc: 0.8627
    Epoch 8/50
    1280/1280 [==============================] - 3289s 3s/step - loss: 0.3339 - tp0.1: 7747141.5660 - fp0.1: 4489700.9196 - tn0.1: 39841447.7486 - fn0.1: 432355.7814 - precision0.1: 0.6332 - recall0.1: 0.9472 - tp0.3: 7357958.1397 - fp0.3:
     2241207.2966 - tn0.3: 42089942.4052 - fn0.3: 821539.2077 - precision0.3: 0.7676 - recall0.3: 0.9001 - tp0.5: 6904184.2022 - fp0.5: 1214222.3841 - tn0.5: 43116930.9828 - fn0.5: 1275313.1452 - precision0.5: 0.8526 - recall0.5: 0.8448 -
    tp0.7: 5918119.9930 - fp0.7: 436485.9844 - tn0.7: 43894677.5480 - fn0.7: 2261377.3544 - precision0.7: 0.9324 - recall0.7: 0.7243 - tp0.9: 4978028.0929 - fp0.9: 136413.7908 - tn0.9: 44194744.8298 - fn0.9: 3201469.2545 - precision0.9: 0.
    9737 - recall0.9: 0.6094 - accuracy: 0.9533 - auc: 0.9664 - val_loss: 0.7192 - val_tp0.1: 3266754.0000 - val_fp0.1: 2261077.0000 - val_tn0.1: 19887774.0000 - val_fn0.1: 798799.0000 - val_precision0.1: 0.5910 - val_recall0.1: 0.8035 - v
    al_tp0.3: 2594715.0000 - val_fp0.3: 1533904.0000 - val_tn0.3: 20614936.0000 - val_fn0.3: 1470838.0000 - val_precision0.3: 0.6285 - val_recall0.3: 0.6382 - val_tp0.5: 2022214.0000 - val_fp0.5: 1204526.0000 - val_tn0.5: 20944316.0000 - v
    al_fn0.5: 2043339.0000 - val_precision0.5: 0.6267 - val_recall0.5: 0.4974 - val_tp0.7: 1633266.0000 - val_fp0.7: 893751.0000 - val_tn0.7: 21255104.0000 - val_fn0.7: 2432287.0000 - val_precision0.7: 0.6463 - val_recall0.7: 0.4017 - val_
    tp0.9: 1072066.0000 - val_fp0.9: 366453.0000 - val_tn0.9: 21782404.0000 - val_fn0.9: 2993487.0000 - val_precision0.9: 0.7453 - val_recall0.9: 0.2637 - val_accuracy: 0.8761 - val_auc: 0.8749
    Epoch 9/50
    1280/1280 [==============================] - 3297s 3s/step - loss: 0.3316 - tp0.1: 7830543.1632 - fp0.1: 4449068.0055 - tn0.1: 39786384.2771 - fn0.1: 444679.2272 - precision0.1: 0.6389 - recall0.1: 0.9467 - tp0.3: 7430622.2623 - fp0.3:
     2239407.0976 - tn0.3: 41996012.4926 - fn0.3: 844600.1280 - precision0.3: 0.7686 - recall0.3: 0.8983 - tp0.5: 6975903.1639 - fp0.5: 1214978.6417 - tn0.5: 43020456.3084 - fn0.5: 1299319.2264 - precision0.5: 0.8518 - recall0.5: 0.8438 -
    tp0.7: 6015111.5628 - fp0.7: 439544.0406 - tn0.7: 43795920.7791 - fn0.7: 2260110.8275 - precision0.7: 0.9325 - recall0.7: 0.7281 - tp0.9: 5071827.3482 - fp0.9: 135401.1327 - tn0.9: 44100020.3169 - fn0.9: 3203395.0422 - precision0.9: 0.
    9746 - recall0.9: 0.6139 - accuracy: 0.9520 - auc: 0.9664 - val_loss: 0.7987 - val_tp0.1: 3338155.0000 - val_fp0.1: 2467457.0000 - val_tn0.1: 19568284.0000 - val_fn0.1: 840502.0000 - val_precision0.1: 0.5750 - val_recall0.1: 0.7989 - v
    al_tp0.3: 2683614.0000 - val_fp0.3: 1892638.0000 - val_tn0.3: 20143100.0000 - val_fn0.3: 1495043.0000 - val_precision0.3: 0.5864 - val_recall0.3: 0.6422 - val_tp0.5: 2192750.0000 - val_fp0.5: 1587985.0000 - val_tn0.5: 20447752.0000 - v
    al_fn0.5: 1985907.0000 - val_precision0.5: 0.5800 - val_recall0.5: 0.5247 - val_tp0.7: 1808320.0000 - val_fp0.7: 1284650.0000 - val_tn0.7: 20751092.0000 - val_fn0.7: 2370337.0000 - val_precision0.7: 0.5847 - val_recall0.7: 0.4328 - val
    _tp0.9: 1198897.0000 - val_fp0.9: 618524.0000 - val_tn0.9: 21417220.0000 - val_fn0.9: 2979760.0000 - val_precision0.9: 0.6597 - val_recall0.9: 0.2869 - val_accuracy: 0.8637 - val_auc: 0.8621
    Epoch 10/50
    1280/1280 [==============================] - 3281s 3s/step - loss: 0.3275 - tp0.1: 7792627.5020 - fp0.1: 4279949.3427 - tn0.1: 40001537.3630 - fn0.1: 436537.2279 - precision0.1: 0.6422 - recall0.1: 0.9472 - tp0.3: 7406705.6987 - fp0.3:
     2160243.7174 - tn0.3: 42121245.1132 - fn0.3: 822459.0312 - precision0.3: 0.7727 - recall0.3: 0.8998 - tp0.5: 6961682.1405 - fp0.5: 1207985.9984 - tn0.5: 43073509.1913 - fn0.5: 1267482.5894 - precision0.5: 0.8511 - recall0.5: 0.8457 -
    tp0.7: 5977782.7900 - fp0.7: 441395.7190 - tn0.7: 43840082.2201 - fn0.7: 2251381.9399 - precision0.7: 0.9300 - recall0.7: 0.7245 - tp0.9: 5047421.8314 - fp0.9: 138119.9415 - tn0.9: 44143384.7346 - fn0.9: 3181742.8985 - precision0.9: 0.
    9728 - recall0.9: 0.6102 - accuracy: 0.9528 - auc: 0.9660 - val_loss: 0.8271 - val_tp0.1: 3497398.0000 - val_fp0.1: 2613010.0000 - val_tn0.1: 19230938.0000 - val_fn0.1: 873051.0000 - val_precision0.1: 0.5724 - val_recall0.1: 0.8002 - v
    al_tp0.3: 2805297.0000 - val_fp0.3: 1928525.0000 - val_tn0.3: 19915426.0000 - val_fn0.3: 1565152.0000 - val_precision0.3: 0.5926 - val_recall0.3: 0.6419 - val_tp0.5: 2250608.0000 - val_fp0.5: 1602046.0000 - val_tn0.5: 20241908.0000 - v
    al_fn0.5: 2119841.0000 - val_precision0.5: 0.5842 - val_recall0.5: 0.5150 - val_tp0.7: 1873937.0000 - val_fp0.7: 1289178.0000 - val_tn0.7: 20554764.0000 - val_fn0.7: 2496512.0000 - val_precision0.7: 0.5924 - val_recall0.7: 0.4288 - val
    _tp0.9: 1282739.0000 - val_fp0.9: 663370.0000 - val_tn0.9: 21180588.0000 - val_fn0.9: 3087710.0000 - val_precision0.9: 0.6591 - val_recall0.9: 0.2935 - val_accuracy: 0.8580 - val_auc: 0.8595
    Epoch 11/50
    1280/1280 [==============================] - 3284s 3s/step - loss: 0.3131 - tp0.1: 7805750.0211 - fp0.1: 4180231.1202 - tn0.1: 40107131.8431 - fn0.1: 417539.0390 - precision0.1: 0.6527 - recall0.1: 0.9499 - tp0.3: 7439055.2615 - fp0.3:
     2100484.2233 - tn0.3: 42186897.5113 - fn0.3: 784233.7986 - precision0.3: 0.7813 - recall0.3: 0.9065 - tp0.5: 7021029.5238 - fp0.5: 1158430.3731 - tn0.5: 43128929.4192 - fn0.5: 1202259.5363 - precision0.5: 0.8590 - recall0.5: 0.8568 -
    tp0.7: 6113168.1468 - fp0.7: 426487.4707 - tn0.7: 43860880.4848 - fn0.7: 2110120.9133 - precision0.7: 0.9352 - recall0.7: 0.7472 - tp0.9: 5208463.6464 - fp0.9: 134634.9001 - tn0.9: 44152728.1069 - fn0.9: 3014825.4137 - precision0.9: 0.
    9752 - recall0.9: 0.6369 - accuracy: 0.9554 - auc: 0.9685 - val_loss: 0.8179 - val_tp0.1: 3171244.0000 - val_fp0.1: 2428671.0000 - val_tn0.1: 19710012.0000 - val_fn0.1: 904466.0000 - val_precision0.1: 0.5663 - val_recall0.1: 0.7781 - v
    al_tp0.3: 2577226.0000 - val_fp0.3: 1780010.0000 - val_tn0.3: 20358686.0000 - val_fn0.3: 1498484.0000 - val_precision0.3: 0.5915 - val_recall0.3: 0.6323 - val_tp0.5: 2103216.0000 - val_fp0.5: 1506964.0000 - val_tn0.5: 20631718.0000 - v
    al_fn0.5: 1972494.0000 - val_precision0.5: 0.5826 - val_recall0.5: 0.5160 - val_tp0.7: 1740648.0000 - val_fp0.7: 1184962.0000 - val_tn0.7: 20953724.0000 - val_fn0.7: 2335062.0000 - val_precision0.7: 0.5950 - val_recall0.7: 0.4271 - val
    _tp0.9: 1152892.0000 - val_fp0.9: 479364.0000 - val_tn0.9: 21659326.0000 - val_fn0.9: 2922818.0000 - val_precision0.9: 0.7063 - val_recall0.9: 0.2829 - val_accuracy: 0.8673 - val_auc: 0.8530
    Epoch 12/50
    1280/1280 [==============================] - 3297s 3s/step - loss: 0.3231 - tp0.1: 7673586.0687 - fp0.1: 4254414.5621 - tn0.1: 40148103.3286 - fn0.1: 434544.6292 - precision0.1: 0.6426 - recall0.1: 0.9456 - tp0.3: 7286525.1889 - fp0.3:
     2133646.5324 - tn0.3: 42268881.5730 - fn0.3: 821605.5090 - precision0.3: 0.7727 - recall0.3: 0.8978 - tp0.5: 6849781.8283 - fp0.5: 1164265.2397 - tn0.5: 43238255.9961 - fn0.5: 1258348.8696 - precision0.5: 0.8542 - recall0.5: 0.8438 -
    tp0.7: 5967163.4153 - fp0.7: 423410.0211 - tn0.7: 43979128.0921 - fn0.7: 2140967.2826 - precision0.7: 0.9348 - recall0.7: 0.7339 - tp0.9: 5093586.0039 - fp0.9: 136848.5098 - tn0.9: 44265710.3138 - fn0.9: 3014544.6940 - precision0.9: 0.
    9745 - recall0.9: 0.6262 - accuracy: 0.9537 - auc: 0.9657 - val_loss: 0.7042 - val_tp0.1: 3495859.0000 - val_fp0.1: 2855125.0000 - val_tn0.1: 19280112.0000 - val_fn0.1: 583301.0000 - val_precision0.1: 0.5504 - val_recall0.1: 0.8570 - v
    al_tp0.3: 2938637.0000 - val_fp0.3: 2173835.0000 - val_tn0.3: 19961412.0000 - val_fn0.3: 1140523.0000 - val_precision0.3: 0.5748 - val_recall0.3: 0.7204 - val_tp0.5: 2404430.0000 - val_fp0.5: 1820758.0000 - val_tn0.5: 20314484.0000 - v
    al_fn0.5: 1674730.0000 - val_precision0.5: 0.5691 - val_recall0.5: 0.5894 - val_tp0.7: 2024378.0000 - val_fp0.7: 1527093.0000 - val_tn0.7: 20608140.0000 - val_fn0.7: 2054782.0000 - val_precision0.7: 0.5700 - val_recall0.7: 0.4963 - val
    _tp0.9: 1381535.0000 - val_fp0.9: 912692.0000 - val_tn0.9: 21222556.0000 - val_fn0.9: 2697625.0000 - val_precision0.9: 0.6022 - val_recall0.9: 0.3387 - val_accuracy: 0.8667 - val_auc: 0.8805
    Epoch 13/50
    1280/1280 [==============================] - 3321s 3s/step - loss: 0.3169 - tp0.1: 7909004.5371 - fp0.1: 4146765.2818 - tn0.1: 40030494.6401 - fn0.1: 424405.8938 - precision0.1: 0.6543 - recall0.1: 0.9500 - tp0.3: 7536643.9977 - fp0.3:
     2085465.0749 - tn0.3: 42091771.2584 - fn0.3: 796766.4333 - precision0.3: 0.7808 - recall0.3: 0.9063 - tp0.5: 7106355.0859 - fp0.5: 1146387.5933 - tn0.5: 43030851.5699 - fn0.5: 1227055.3450 - precision0.5: 0.8590 - recall0.5: 0.8551 -
    tp0.7: 6229305.1702 - fp0.7: 429169.7603 - tn0.7: 43748120.6526 - fn0.7: 2104105.2607 - precision0.7: 0.9350 - recall0.7: 0.7476 - tp0.9: 5365662.5207 - fp0.9: 142482.5558 - tn0.9: 44034765.0882 - fn0.9: 2967747.9102 - precision0.9: 0.
    9736 - recall0.9: 0.6436 - accuracy: 0.9544 - auc: 0.9678 - val_loss: 0.8398 - val_tp0.1: 3245885.0000 - val_fp0.1: 2536432.0000 - val_tn0.1: 19519276.0000 - val_fn0.1: 912810.0000 - val_precision0.1: 0.5613 - val_recall0.1: 0.7805 - v
    al_tp0.3: 2564525.0000 - val_fp0.3: 1814112.0000 - val_tn0.3: 20241596.0000 - val_fn0.3: 1594170.0000 - val_precision0.3: 0.5857 - val_recall0.3: 0.6167 - val_tp0.5: 2015465.0000 - val_fp0.5: 1451082.0000 - val_tn0.5: 20604624.0000 - v
    al_fn0.5: 2143230.0000 - val_precision0.5: 0.5814 - val_recall0.5: 0.4846 - val_tp0.7: 1621561.0000 - val_fp0.7: 1053037.0000 - val_tn0.7: 21002666.0000 - val_fn0.7: 2537134.0000 - val_precision0.7: 0.6063 - val_recall0.7: 0.3899 - val
    _tp0.9: 1065568.0000 - val_fp0.9: 460390.0000 - val_tn0.9: 21595312.0000 - val_fn0.9: 3093127.0000 - val_precision0.9: 0.6983 - val_recall0.9: 0.2562 - val_accuracy: 0.8629 - val_auc: 0.8493
    Epoch 14/50
    1280/1280 [==============================] - 3314s 3s/step - loss: 0.3076 - tp0.1: 7765395.1202 - fp0.1: 4130752.8431 - tn0.1: 40213352.2810 - fn0.1: 401143.6737 - precision0.1: 0.6501 - recall0.1: 0.9503 - tp0.3: 7389168.6144 - fp0.3:
     2041484.8189 - tn0.3: 42302624.6870 - fn0.3: 777370.1795 - precision0.3: 0.7825 - recall0.3: 0.9030 - tp0.5: 6961617.5222 - fp0.5: 1091504.5597 - tn0.5: 43252606.1101 - fn0.5: 1204921.2717 - precision0.5: 0.8642 - recall0.5: 0.8492 -
    tp0.7: 6118152.3286 - fp0.7: 416044.1280 - tn0.7: 43928097.5558 - fn0.7: 2048386.4653 - precision0.7: 0.9364 - recall0.7: 0.7464 - tp0.9: 5275694.6081 - fp0.9: 131773.3169 - tn0.9: 44212345.9774 - fn0.9: 2890844.1858 - precision0.9: 0.
    9759 - recall0.9: 0.6429 - accuracy: 0.9559 - auc: 0.9681 - val_loss: 0.7646 - val_tp0.1: 3394116.0000 - val_fp0.1: 2434359.0000 - val_tn0.1: 19620386.0000 - val_fn0.1: 765541.0000 - val_precision0.1: 0.5823 - val_recall0.1: 0.8160 - v
    al_tp0.3: 2854698.0000 - val_fp0.3: 1867547.0000 - val_tn0.3: 20187194.0000 - val_fn0.3: 1304959.0000 - val_precision0.3: 0.6045 - val_recall0.3: 0.6863 - val_tp0.5: 2392164.0000 - val_fp0.5: 1600876.0000 - val_tn0.5: 20453860.0000 - v
    al_fn0.5: 1767493.0000 - val_precision0.5: 0.5991 - val_recall0.5: 0.5751 - val_tp0.7: 2014352.0000 - val_fp0.7: 1325938.0000 - val_tn0.7: 20728804.0000 - val_fn0.7: 2145305.0000 - val_precision0.7: 0.6030 - val_recall0.7: 0.4843 - val
    _tp0.9: 1382581.0000 - val_fp0.9: 710524.0000 - val_tn0.9: 21344216.0000 - val_fn0.9: 2777076.0000 - val_precision0.9: 0.6605 - val_recall0.9: 0.3324 - val_accuracy: 0.8715 - val_auc: 0.8680
    Epoch 15/50
    1280/1280 [==============================] - 3321s 3s/step - loss: 0.3012 - tp0.1: 7918202.5168 - fp0.1: 3944007.1944 - tn0.1: 40234677.5113 - fn0.1: 413802.1538 - precision0.1: 0.6678 - recall0.1: 0.9509 - tp0.3: 7545589.1913 - fp0.3:
     1942385.4692 - tn0.3: 42236259.0687 - fn0.3: 786415.4793 - precision0.3: 0.7966 - recall0.3: 0.9059 - tp0.5: 7139685.9657 - fp0.5: 1075216.9391 - tn0.5: 43103450.5808 - fn0.5: 1192318.7049 - precision0.5: 0.8712 - recall0.5: 0.8570 -
    tp0.7: 6377161.2748 - fp0.7: 450450.9469 - tn0.7: 43728240.0265 - fn0.7: 1954843.3958 - precision0.7: 0.9369 - recall0.7: 0.7641 - tp0.9: 5506362.6120 - fp0.9: 143446.8002 - tn0.9: 44035203.3466 - fn0.9: 2825642.0585 - precision0.9: 0.
    9752 - recall0.9: 0.6625 - accuracy: 0.9571 - auc: 0.9692 - val_loss: 0.7740 - val_tp0.1: 3355047.0000 - val_fp0.1: 2511597.0000 - val_tn0.1: 19571600.0000 - val_fn0.1: 776157.0000 - val_precision0.1: 0.5719 - val_recall0.1: 0.8121 - v
    al_tp0.3: 2793785.0000 - val_fp0.3: 1928920.0000 - val_tn0.3: 20154280.0000 - val_fn0.3: 1337419.0000 - val_precision0.3: 0.5916 - val_recall0.3: 0.6763 - val_tp0.5: 2272116.0000 - val_fp0.5: 1578770.0000 - val_tn0.5: 20504436.0000 - v
    al_fn0.5: 1859088.0000 - val_precision0.5: 0.5900 - val_recall0.5: 0.5500 - val_tp0.7: 1883335.0000 - val_fp0.7: 1254872.0000 - val_tn0.7: 20828326.0000 - val_fn0.7: 2247869.0000 - val_precision0.7: 0.6001 - val_recall0.7: 0.4559 - val
    _tp0.9: 1294491.0000 - val_fp0.9: 656247.0000 - val_tn0.9: 21426948.0000 - val_fn0.9: 2836713.0000 - val_precision0.9: 0.6636 - val_recall0.9: 0.3133 - val_accuracy: 0.8689 - val_auc: 0.8643
    Epoch 16/50
    1280/1280 [==============================] - 3322s 3s/step - loss: 0.2996 - tp0.1: 7939368.5972 - fp0.1: 3969169.3778 - tn0.1: 40211699.6347 - fn0.1: 390406.3911 - precision0.1: 0.6606 - recall0.1: 0.9528 - tp0.3: 7573992.7580 - fp0.3:
     1993073.9243 - tn0.3: 42187811.5020 - fn0.3: 755782.2303 - precision0.3: 0.7866 - recall0.3: 0.9074 - tp0.5: 7177194.8970 - fp0.5: 1109584.0390 - tn0.5: 43071285.9805 - fn0.5: 1152580.0913 - precision0.5: 0.8632 - recall0.5: 0.8586 -
    tp0.7: 6618657.6151 - fp0.7: 578836.1054 - tn0.7: 43602043.0445 - fn0.7: 1711117.3731 - precision0.7: 0.9188 - recall0.7: 0.7896 - tp0.9: 5482513.6440 - fp0.9: 141140.2100 - tn0.9: 44039753.1483 - fn0.9: 2847261.3443 - precision0.9: 0.
    9744 - recall0.9: 0.6538 - accuracy: 0.9565 - auc: 0.9693 - val_loss: 0.8054 - val_tp0.1: 3345481.0000 - val_fp0.1: 2254991.0000 - val_tn0.1: 19696408.0000 - val_fn0.1: 917515.0000 - val_precision0.1: 0.5974 - val_recall0.1: 0.7848 - v
    al_tp0.3: 2726884.0000 - val_fp0.3: 1640058.0000 - val_tn0.3: 20311344.0000 - val_fn0.3: 1536112.0000 - val_precision0.3: 0.6244 - val_recall0.3: 0.6397 - val_tp0.5: 2263115.0000 - val_fp0.5: 1402171.0000 - val_tn0.5: 20549236.0000 - v
    al_fn0.5: 1999881.0000 - val_precision0.5: 0.6174 - val_recall0.5: 0.5309 - val_tp0.7: 1906416.0000 - val_fp0.7: 1106825.0000 - val_tn0.7: 20844576.0000 - val_fn0.7: 2356580.0000 - val_precision0.7: 0.6327 - val_recall0.7: 0.4472 - val
    _tp0.9: 1297258.0000 - val_fp0.9: 495212.0000 - val_tn0.9: 21456192.0000 - val_fn0.9: 2965738.0000 - val_precision0.9: 0.7237 - val_recall0.9: 0.3043 - val_accuracy: 0.8702 - val_auc: 0.8614
    Epoch 17/50
    1280/1280 [==============================] - 3281s 3s/step - loss: 0.2900 - tp0.1: 7724454.9820 - fp0.1: 3793475.2631 - tn0.1: 40600766.8860 - fn0.1: 391947.2693 - precision0.1: 0.6695 - recall0.1: 0.9516 - tp0.3: 7368504.9789 - fp0.3:
     1879500.1913 - tn0.3: 42514764.5855 - fn0.3: 747897.2724 - precision0.3: 0.7960 - recall0.3: 0.9078 - tp0.5: 6984976.5464 - fp0.5: 1046817.2740 - tn0.5: 43347456.7892 - fn0.5: 1131425.7049 - precision0.5: 0.8690 - recall0.5: 0.8606 -
    tp0.7: 6474243.4176 - fp0.7: 552184.5105 - tn0.7: 43842053.0726 - fn0.7: 1642158.8337 - precision0.7: 0.9206 - recall0.7: 0.7983 - tp0.9: 5438164.0788 - fp0.9: 137406.8056 - tn0.9: 44256829.4044 - fn0.9: 2678238.1725 - precision0.9: 0.
    9752 - recall0.9: 0.6718 - accuracy: 0.9584 - auc: 0.9698 - val_loss: 0.8475 - val_tp0.1: 3212091.0000 - val_fp0.1: 2556630.0000 - val_tn0.1: 19565418.0000 - val_fn0.1: 880263.0000 - val_precision0.1: 0.5568 - val_recall0.1: 0.7849 - v
    al_tp0.3: 2676698.0000 - val_fp0.3: 1975975.0000 - val_tn0.3: 20146070.0000 - val_fn0.3: 1415656.0000 - val_precision0.3: 0.5753 - val_recall0.3: 0.6541 - val_tp0.5: 2228667.0000 - val_fp0.5: 1724443.0000 - val_tn0.5: 20397608.0000 - v
    al_fn0.5: 1863687.0000 - val_precision0.5: 0.5638 - val_recall0.5: 0.5446 - val_tp0.7: 1876988.0000 - val_fp0.7: 1421139.0000 - val_tn0.7: 20700906.0000 - val_fn0.7: 2215366.0000 - val_precision0.7: 0.5691 - val_recall0.7: 0.4587 - val
    _tp0.9: 1318523.0000 - val_fp0.9: 744640.0000 - val_tn0.9: 21377414.0000 - val_fn0.9: 2773831.0000 - val_precision0.9: 0.6391 - val_recall0.9: 0.3222 - val_accuracy: 0.8631 - val_auc: 0.8479
    Epoch 18/50
    1280/1280 [==============================] - 3291s 3s/step - loss: 0.2960 - tp0.1: 7878258.3583 - fp0.1: 3966414.4723 - tn0.1: 40266792.8720 - fn0.1: 399191.5441 - precision0.1: 0.6660 - recall0.1: 0.9518 - tp0.3: 7519728.0133 - fp0.3:
     1997387.8189 - tn0.3: 42235815.4325 - fn0.3: 757721.8891 - precision0.3: 0.7910 - recall0.3: 0.9083 - tp0.5: 7117268.5543 - fp0.5: 1103594.4309 - tn0.5: 43129603.3357 - fn0.5: 1160181.3482 - precision0.5: 0.8660 - recall0.5: 0.8598 -
    tp0.7: 6559720.6885 - fp0.7: 569817.0601 - tn0.7: 43663373.8767 - fn0.7: 1717729.2139 - precision0.7: 0.9203 - recall0.7: 0.7929 - tp0.9: 5460392.9578 - fp0.9: 133811.0039 - tn0.9: 44099404.8353 - fn0.9: 2817056.9446 - precision0.9: 0.
    9765 - recall0.9: 0.6620 - accuracy: 0.9569 - auc: 0.9692 - val_loss: 2.4700 - val_tp0.1: 3478680.0000 - val_fp0.1: 2735141.0000 - val_tn0.1: 19398832.0000 - val_fn0.1: 601742.0000 - val_precision0.1: 0.5598 - val_recall0.1: 0.8525 - v
    al_tp0.3: 2990828.0000 - val_fp0.3: 2147854.0000 - val_tn0.3: 19986120.0000 - val_fn0.3: 1089594.0000 - val_precision0.3: 0.5820 - val_recall0.3: 0.7330 - val_tp0.5: 2523597.0000 - val_fp0.5: 1960329.0000 - val_tn0.5: 20173640.0000 - v
    al_fn0.5: 1556825.0000 - val_precision0.5: 0.5628 - val_recall0.5: 0.6185 - val_tp0.7: 2211753.0000 - val_fp0.7: 1837061.0000 - val_tn0.7: 20296910.0000 - val_fn0.7: 1868669.0000 - val_precision0.7: 0.5463 - val_recall0.7: 0.5420 - val
    _tp0.9: 1781271.0000 - val_fp0.9: 1591848.0000 - val_tn0.9: 20542120.0000 - val_fn0.9: 2299151.0000 - val_precision0.9: 0.5281 - val_recall0.9: 0.4365 - val_accuracy: 0.8658 - val_auc: 0.8707
    Epoch 19/50
    1280/1280 [==============================] - 3287s 3s/step - loss: 0.2909 - tp0.1: 7989303.7198 - fp0.1: 3862557.4653 - tn0.1: 40273532.0609 - fn0.1: 385255.3396 - precision0.1: 0.6759 - recall0.1: 0.9542 - tp0.3: 7630793.7799 - fp0.3:
     1926931.7330 - tn0.3: 42209171.3419 - fn0.3: 743765.2795 - precision0.3: 0.7994 - recall0.3: 0.9113 - tp0.5: 7240582.7869 - fp0.5: 1067331.3224 - tn0.5: 43068761.2030 - fn0.5: 1133976.2724 - precision0.5: 0.8729 - recall0.5: 0.8645 -
    tp0.7: 6700153.0500 - fp0.7: 553561.1038 - tn0.7: 43582525.7065 - fn0.7: 1674406.0094 - precision0.7: 0.9247 - recall0.7: 0.7993 - tp0.9: 5619106.6877 - fp0.9: 138913.2576 - tn0.9: 43997191.2326 - fn0.9: 2755452.3716 - precision0.9: 0.
    9761 - recall0.9: 0.6706 - accuracy: 0.9580 - auc: 0.9708 - val_loss: 0.8749 - val_tp0.1: 3092711.0000 - val_fp0.1: 2067604.0000 - val_tn0.1: 20017088.0000 - val_fn0.1: 1036989.0000 - val_precision0.1: 0.5993 - val_recall0.1: 0.7489 -
    val_tp0.3: 2486761.0000 - val_fp0.3: 1549670.0000 - val_tn0.3: 20535032.0000 - val_fn0.3: 1642939.0000 - val_precision0.3: 0.6161 - val_recall0.3: 0.6022 - val_tp0.5: 2068354.0000 - val_fp0.5: 1334103.0000 - val_tn0.5: 20750600.0000 -
    val_fn0.5: 2061346.0000 - val_precision0.5: 0.6079 - val_recall0.5: 0.5008 - val_tp0.7: 1728814.0000 - val_fp0.7: 1041223.0000 - val_tn0.7: 21043472.0000 - val_fn0.7: 2400886.0000 - val_precision0.7: 0.6241 - val_recall0.7: 0.4186 - va
    l_tp0.9: 1199934.0000 - val_fp0.9: 520759.0000 - val_tn0.9: 21563940.0000 - val_fn0.9: 2929766.0000 - val_precision0.9: 0.6974 - val_recall0.9: 0.2906 - val_accuracy: 0.8705 - val_auc: 0.8436
    Epoch 20/50
    1280/1280 [==============================] - 3278s 3s/step - loss: 0.2874 - tp0.1: 7845801.3482 - fp0.1: 3747571.3029 - tn0.1: 40520331.9508 - fn0.1: 396960.4161 - precision0.1: 0.6782 - recall0.1: 0.9514 - tp0.3: 7499855.0742 - fp0.3:
     1875299.3388 - tn0.3: 42392596.7424 - fn0.3: 742906.6901 - precision0.3: 0.8017 - recall0.3: 0.9098 - tp0.5: 7121660.4918 - fp0.5: 1042386.0788 - tn0.5: 43225501.8384 - fn0.5: 1121101.2724 - precision0.5: 0.8739 - recall0.5: 0.8640 -
    tp0.7: 6592531.3404 - fp0.7: 538088.3911 - tn0.7: 43729809.0500 - fn0.7: 1650230.4239 - precision0.7: 0.9254 - recall0.7: 0.8003 - tp0.9: 5543415.5105 - fp0.9: 131938.6378 - tn0.9: 44135942.6190 - fn0.9: 2699346.2537 - precision0.9: 0.
    9770 - recall0.9: 0.6728 - accuracy: 0.9592 - auc: 0.9698 - val_loss: 0.9565 - val_tp0.1: 2943258.0000 - val_fp0.1: 2086336.0000 - val_tn0.1: 20045976.0000 - val_fn0.1: 1138833.0000 - val_precision0.1: 0.5852 - val_recall0.1: 0.7210 -
    val_tp0.3: 2419480.0000 - val_fp0.3: 1662989.0000 - val_tn0.3: 20469314.0000 - val_fn0.3: 1662611.0000 - val_precision0.3: 0.5927 - val_recall0.3: 0.5927 - val_tp0.5: 2018052.0000 - val_fp0.5: 1478186.0000 - val_tn0.5: 20654124.0000 -
    val_fn0.5: 2064039.0000 - val_precision0.5: 0.5772 - val_recall0.5: 0.4944 - val_tp0.7: 1715248.0000 - val_fp0.7: 1231354.0000 - val_tn0.7: 20900956.0000 - val_fn0.7: 2366843.0000 - val_precision0.7: 0.5821 - val_recall0.7: 0.4202 - va
    l_tp0.9: 1245570.0000 - val_fp0.9: 677868.0000 - val_tn0.9: 21454434.0000 - val_fn0.9: 2836521.0000 - val_precision0.9: 0.6476 - val_recall0.9: 0.3051 - val_accuracy: 0.8649 - val_auc: 0.8275
    Epoch 21/50
    1280/1280 [==============================] - 3290s 3s/step - loss: 0.2804 - tp0.1: 7821343.4738 - fp0.1: 3664726.1171 - tn0.1: 40657232.2397 - fn0.1: 367350.5347 - precision0.1: 0.6815 - recall0.1: 0.9542 - tp0.3: 7489144.7580 - fp0.3:
     1868735.6292 - tn0.3: 42453227.8447 - fn0.3: 699549.2506 - precision0.3: 0.8010 - recall0.3: 0.9132 - tp0.5: 7115665.6729 - fp0.5: 1043113.8876 - tn0.5: 43278844.9688 - fn0.5: 1073028.3357 - precision0.5: 0.8724 - recall0.5: 0.8679 -
    tp0.7: 6602597.8665 - fp0.7: 551623.0804 - tn0.7: 43770320.6628 - fn0.7: 1586096.1421 - precision0.7: 0.9226 - recall0.7: 0.8053 - tp0.9: 5554093.7018 - fp0.9: 134091.5519 - tn0.9: 44187854.0671 - fn0.9: 2634600.3068 - precision0.9: 0.
    9763 - recall0.9: 0.6772 - accuracy: 0.9595 - auc: 0.9714 - val_loss: 0.8160 - val_tp0.1: 3192064.0000 - val_fp0.1: 2182394.0000 - val_tn0.1: 19900398.0000 - val_fn0.1: 939537.0000 - val_precision0.1: 0.5939 - val_recall0.1: 0.7726 - v
    al_tp0.3: 2628251.0000 - val_fp0.3: 1676252.0000 - val_tn0.3: 20406544.0000 - val_fn0.3: 1503350.0000 - val_precision0.3: 0.6106 - val_recall0.3: 0.6361 - val_tp0.5: 2262964.0000 - val_fp0.5: 1459711.0000 - val_tn0.5: 20623092.0000 - v
    al_fn0.5: 1868637.0000 - val_precision0.5: 0.6079 - val_recall0.5: 0.5477 - val_tp0.7: 1912004.0000 - val_fp0.7: 1114695.0000 - val_tn0.7: 20968108.0000 - val_fn0.7: 2219597.0000 - val_precision0.7: 0.6317 - val_recall0.7: 0.4628 - val
    _tp0.9: 1266412.0000 - val_fp0.9: 537157.0000 - val_tn0.9: 21545640.0000 - val_fn0.9: 2865189.0000 - val_precision0.9: 0.7022 - val_recall0.9: 0.3065 - val_accuracy: 0.8730 - val_auc: 0.8570
    Epoch 22/50
    1280/1280 [==============================] - 3337s 3s/step - loss: 0.2695 - tp0.1: 7964599.1304 - fp0.1: 3487980.5644 - tn0.1: 40692123.6760 - fn0.1: 365971.2584 - precision0.1: 0.6952 - recall0.1: 0.9559 - tp0.3: 7639300.1928 - fp0.3:
     1768977.4333 - tn0.3: 42411101.8767 - fn0.3: 691270.1959 - precision0.3: 0.8119 - recall0.3: 0.9169 - tp0.5: 7283022.7681 - fp0.5: 1008769.9313 - tn0.5: 43171283.5621 - fn0.5: 1047547.6206 - precision0.5: 0.8781 - recall0.5: 0.8740 -
    tp0.7: 6797706.8946 - fp0.7: 543031.7963 - tn0.7: 43637044.5636 - fn0.7: 1532863.4941 - precision0.7: 0.9257 - recall0.7: 0.8158 - tp0.9: 5748924.9516 - fp0.9: 136050.2350 - tn0.9: 44044056.0921 - fn0.9: 2581645.4372 - precision0.9: 0.
    9771 - recall0.9: 0.6893 - accuracy: 0.9609 - auc: 0.9731 - val_loss: 0.8131 - val_tp0.1: 3187368.0000 - val_fp0.1: 2205558.0000 - val_tn0.1: 19939310.0000 - val_fn0.1: 882169.0000 - val_precision0.1: 0.5910 - val_recall0.1: 0.7832 - v
    al_tp0.3: 2568349.0000 - val_fp0.3: 1630822.0000 - val_tn0.3: 20514044.0000 - val_fn0.3: 1501188.0000 - val_precision0.3: 0.6116 - val_recall0.3: 0.6311 - val_tp0.5: 2132479.0000 - val_fp0.5: 1391371.0000 - val_tn0.5: 20753492.0000 - v
    al_fn0.5: 1937058.0000 - val_precision0.5: 0.6052 - val_recall0.5: 0.5240 - val_tp0.7: 1724396.0000 - val_fp0.7: 1052743.0000 - val_tn0.7: 21092120.0000 - val_fn0.7: 2345141.0000 - val_precision0.7: 0.6209 - val_recall0.7: 0.4237 - val
    _tp0.9: 1088044.0000 - val_fp0.9: 509365.0000 - val_tn0.9: 21635508.0000 - val_fn0.9: 2981493.0000 - val_precision0.9: 0.6811 - val_recall0.9: 0.2674 - val_accuracy: 0.8730 - val_auc: 0.8564
    Epoch 23/50
    1280/1280 [==============================] - 3288s 3s/step - loss: 0.2689 - tp0.1: 7750995.1733 - fp0.1: 3597605.4887 - tn0.1: 40799177.1624 - fn0.1: 362898.6737 - precision0.1: 0.6830 - recall0.1: 0.9562 - tp0.3: 7415912.0773 - fp0.3:
     1777617.9633 - tn0.3: 42619150.5051 - fn0.3: 697981.7697 - precision0.3: 0.8063 - recall0.3: 0.9146 - tp0.5: 7059667.4192 - fp0.5: 992741.5051 - tn0.5: 43404008.9727 - fn0.5: 1054226.4278 - precision0.5: 0.8769 - recall0.5: 0.8705 - t
    p0.7: 6579822.4957 - fp0.7: 525915.9500 - tn0.7: 43870813.2443 - fn0.7: 1534071.3513 - precision0.7: 0.9263 - recall0.7: 0.8112 - tp0.9: 5573469.2935 - fp0.9: 125489.0593 - tn0.9: 44271266.5379 - fn0.9: 2540424.5535 - precision0.9: 0.9
    785 - recall0.9: 0.6878 - accuracy: 0.9612 - auc: 0.9727 - val_loss: 0.8991 - val_tp0.1: 3067116.0000 - val_fp0.1: 2378685.0000 - val_tn0.1: 19772492.0000 - val_fn0.1: 996109.0000 - val_precision0.1: 0.5632 - val_recall0.1: 0.7548 - va
    l_tp0.3: 2525982.0000 - val_fp0.3: 1869663.0000 - val_tn0.3: 20281514.0000 - val_fn0.3: 1537243.0000 - val_precision0.3: 0.5747 - val_recall0.3: 0.6217 - val_tp0.5: 2090818.0000 - val_fp0.5: 1570273.0000 - val_tn0.5: 20580908.0000 - va
    l_fn0.5: 1972407.0000 - val_precision0.5: 0.5711 - val_recall0.5: 0.5146 - val_tp0.7: 1691191.0000 - val_fp0.7: 1177904.0000 - val_tn0.7: 20973274.0000 - val_fn0.7: 2372034.0000 - val_precision0.7: 0.5895 - val_recall0.7: 0.4162 - val_
    tp0.9: 1101527.0000 - val_fp0.9: 574543.0000 - val_tn0.9: 21576624.0000 - val_fn0.9: 2961698.0000 - val_precision0.9: 0.6572 - val_recall0.9: 0.2711 - val_accuracy: 0.8649 - val_auc: 0.8377
    Epoch 24/50
    1280/1280 [==============================] - 3284s 3s/step - loss: 0.2805 - tp0.1: 7918656.6589 - fp0.1: 3592495.9235 - tn0.1: 40635399.4606 - fn0.1: 364123.3185 - precision0.1: 0.6839 - recall0.1: 0.9541 - tp0.3: 7581799.2061 - fp0.3:
     1810020.2311 - tn0.3: 42417852.9594 - fn0.3: 700980.7713 - precision0.3: 0.8032 - recall0.3: 0.9122 - tp0.5: 7211523.7502 - fp0.5: 1027547.9930 - tn0.5: 43200324.1296 - fn0.5: 1071256.2272 - precision0.5: 0.8718 - recall0.5: 0.8665 -
    tp0.7: 6727091.3692 - fp0.7: 558955.8712 - tn0.7: 43668944.4075 - fn0.7: 1555688.6081 - precision0.7: 0.9209 - recall0.7: 0.8072 - tp0.9: 5696121.9555 - fp0.9: 142622.7455 - tn0.9: 44085280.0343 - fn0.9: 2586658.0219 - precision0.9: 0.
    9746 - recall0.9: 0.6826 - accuracy: 0.9591 - auc: 0.9708 - val_loss: 0.9386 - val_tp0.1: 2779932.0000 - val_fp0.1: 1939286.0000 - val_tn0.1: 20344936.0000 - val_fn0.1: 1150243.0000 - val_precision0.1: 0.5891 - val_recall0.1: 0.7073 -
    val_tp0.3: 2181594.0000 - val_fp0.3: 1457307.0000 - val_tn0.3: 20826908.0000 - val_fn0.3: 1748581.0000 - val_precision0.3: 0.5995 - val_recall0.3: 0.5551 - val_tp0.5: 1772386.0000 - val_fp0.5: 1197130.0000 - val_tn0.5: 21087092.0000 -
    val_fn0.5: 2157789.0000 - val_precision0.5: 0.5969 - val_recall0.5: 0.4510 - val_tp0.7: 1397374.0000 - val_fp0.7: 840833.0000 - val_tn0.7: 21443384.0000 - val_fn0.7: 2532801.0000 - val_precision0.7: 0.6243 - val_recall0.7: 0.3556 - val
    _tp0.9: 891892.0000 - val_fp0.9: 338054.0000 - val_tn0.9: 21946184.0000 - val_fn0.9: 3038283.0000 - val_precision0.9: 0.7251 - val_recall0.9: 0.2269 - val_accuracy: 0.8720 - val_auc: 0.8227
    Epoch 25/50
    1280/1280 [==============================] - 3312s 3s/step - loss: 0.2614 - tp0.1: 7920216.7338 - fp0.1: 3526618.6542 - tn0.1: 40723751.9001 - fn0.1: 340058.4512 - precision0.1: 0.6948 - recall0.1: 0.9594 - tp0.3: 7591746.0851 - fp0.3:
     1770255.0687 - tn0.3: 42480125.7884 - fn0.3: 668529.0999 - precision0.3: 0.8133 - recall0.3: 0.9199 - tp0.5: 7232148.8064 - fp0.5: 1004410.7486 - tn0.5: 43245980.8556 - fn0.5: 1028126.3786 - precision0.5: 0.8800 - recall0.5: 0.8766 -
    tp0.7: 6751684.1514 - fp0.7: 540161.4325 - tn0.7: 43710214.4629 - fn0.7: 1508591.0336 - precision0.7: 0.9275 - recall0.7: 0.8179 - tp0.9: 5682987.5933 - fp0.9: 133426.7330 - tn0.9: 44116927.1835 - fn0.9: 2577287.5917 - precision0.9: 0.
    9781 - recall0.9: 0.6873 - accuracy: 0.9617 - auc: 0.9742 - val_loss: 0.7990 - val_tp0.1: 3208335.0000 - val_fp0.1: 2291707.0000 - val_tn0.1: 19888704.0000 - val_fn0.1: 825651.0000 - val_precision0.1: 0.5833 - val_recall0.1: 0.7953 - v
    al_tp0.3: 2646324.0000 - val_fp0.3: 1812815.0000 - val_tn0.3: 20367606.0000 - val_fn0.3: 1387662.0000 - val_precision0.3: 0.5935 - val_recall0.3: 0.6560 - val_tp0.5: 2184535.0000 - val_fp0.5: 1449172.0000 - val_tn0.5: 20731252.0000 - v
    al_fn0.5: 1849451.0000 - val_precision0.5: 0.6012 - val_recall0.5: 0.5415 - val_tp0.7: 1700807.0000 - val_fp0.7: 1059055.0000 - val_tn0.7: 21121356.0000 - val_fn0.7: 2333179.0000 - val_precision0.7: 0.6163 - val_recall0.7: 0.4216 - val
    _tp0.9: 1000799.0000 - val_fp0.9: 603281.0000 - val_tn0.9: 21577132.0000 - val_fn0.9: 3033187.0000 - val_precision0.9: 0.6239 - val_recall0.9: 0.2481 - val_accuracy: 0.8742 - val_auc: 0.8559
    Epoch 26/50
    1280/1280 [==============================] - 3292s 3s/step - loss: 0.2564 - tp0.1: 7951938.6698 - fp0.1: 3507608.0250 - tn0.1: 40722855.8618 - fn0.1: 328256.7658 - precision0.1: 0.6956 - recall0.1: 0.9610 - tp0.3: 7628245.6300 - fp0.3:
     1764444.8579 - tn0.3: 42466010.5285 - fn0.3: 651949.8056 - precision0.3: 0.8130 - recall0.3: 0.9222 - tp0.5: 7277018.4473 - fp0.5: 993563.1429 - tn0.5: 43236890.8002 - fn0.5: 1003176.9883 - precision0.5: 0.8801 - recall0.5: 0.8802 - t
    p0.7: 6809211.9329 - fp0.7: 538938.6565 - tn0.7: 43691507.5847 - fn0.7: 1470983.5027 - precision0.7: 0.9271 - recall0.7: 0.8247 - tp0.9: 5778468.3435 - fp0.9: 130088.7877 - tn0.9: 44100381.7361 - fn0.9: 2501727.0921 - precision0.9: 0.9
    783 - recall0.9: 0.7006 - accuracy: 0.9620 - auc: 0.9752 - val_loss: 0.9080 - val_tp0.1: 2917153.0000 - val_fp0.1: 2217207.0000 - val_tn0.1: 20036056.0000 - val_fn0.1: 1043985.0000 - val_precision0.1: 0.5682 - val_recall0.1: 0.7364 - v
    al_tp0.3: 2324132.0000 - val_fp0.3: 1709552.0000 - val_tn0.3: 20543718.0000 - val_fn0.3: 1637006.0000 - val_precision0.3: 0.5762 - val_recall0.3: 0.5867 - val_tp0.5: 1885949.0000 - val_fp0.5: 1373092.0000 - val_tn0.5: 20880176.0000 - v
    al_fn0.5: 2075189.0000 - val_precision0.5: 0.5787 - val_recall0.5: 0.4761 - val_tp0.7: 1470934.0000 - val_fp0.7: 951612.0000 - val_tn0.7: 21301648.0000 - val_fn0.7: 2490204.0000 - val_precision0.7: 0.6072 - val_recall0.7: 0.3713 - val_
    tp0.9: 925487.0000 - val_fp0.9: 440991.0000 - val_tn0.9: 21812280.0000 - val_fn0.9: 3035651.0000 - val_precision0.9: 0.6773 - val_recall0.9: 0.2336 - val_accuracy: 0.8685 - val_auc: 0.8286
    Epoch 27/50
    1280/1280 [==============================] - 3294s 3s/step - loss: 0.2645 - tp0.1: 8005473.6409 - fp0.1: 3575769.4098 - tn0.1: 40581931.1319 - fn0.1: 347472.6667 - precision0.1: 0.6907 - recall0.1: 0.9587 - tp0.3: 7670517.9243 - fp0.3:
     1798604.7674 - tn0.3: 42359111.5941 - fn0.3: 682428.3833 - precision0.3: 0.8108 - recall0.3: 0.9185 - tp0.5: 7312500.5558 - fp0.5: 1024504.0812 - tn0.5: 43133203.4434 - fn0.5: 1040445.7518 - precision0.5: 0.8784 - recall0.5: 0.8764 -
    tp0.7: 6846122.2201 - fp0.7: 554274.4294 - tn0.7: 43603409.3521 - fn0.7: 1506824.0874 - precision0.7: 0.9264 - recall0.7: 0.8215 - tp0.9: 5804102.2420 - fp0.9: 127947.9672 - tn0.9: 44029764.9969 - fn0.9: 2548844.0656 - precision0.9: 0.
    9789 - recall0.9: 0.6991 - accuracy: 0.9613 - auc: 0.9737 - val_loss: 0.7467 - val_tp0.1: 3265250.0000 - val_fp0.1: 2199966.0000 - val_tn0.1: 19968754.0000 - val_fn0.1: 780424.0000 - val_precision0.1: 0.5975 - val_recall0.1: 0.8071 - v
    al_tp0.3: 2595986.0000 - val_fp0.3: 1569977.0000 - val_tn0.3: 20598752.0000 - val_fn0.3: 1449688.0000 - val_precision0.3: 0.6231 - val_recall0.3: 0.6417 - val_tp0.5: 2090775.0000 - val_fp0.5: 1320764.0000 - val_tn0.5: 20847968.0000 - v
    al_fn0.5: 1954899.0000 - val_precision0.5: 0.6129 - val_recall0.5: 0.5168 - val_tp0.7: 1610859.0000 - val_fp0.7: 895071.0000 - val_tn0.7: 21273662.0000 - val_fn0.7: 2434815.0000 - val_precision0.7: 0.6428 - val_recall0.7: 0.3982 - val_
    tp0.9: 934620.0000 - val_fp0.9: 380349.0000 - val_tn0.9: 21788384.0000 - val_fn0.9: 3111054.0000 - val_precision0.9: 0.7108 - val_recall0.9: 0.2310 - val_accuracy: 0.8750 - val_auc: 0.8659
    Epoch 28/50
    1280/1280 [==============================] - 3286s 3s/step - loss: 0.2563 - tp0.1: 7900461.9563 - fp0.1: 3575374.1405 - tn0.1: 40711544.3599 - fn0.1: 323306.9867 - precision0.1: 0.6896 - recall0.1: 0.9612 - tp0.3: 7567966.7549 - fp0.3:
     1768843.2389 - tn0.3: 42518056.5605 - fn0.3: 655802.1881 - precision0.3: 0.8113 - recall0.3: 0.9206 - tp0.5: 7202561.9477 - fp0.5: 988935.1319 - tn0.5: 43297946.7713 - fn0.5: 1021206.9953 - precision0.5: 0.8795 - recall0.5: 0.8766 - t
    p0.7: 6725537.1077 - fp0.7: 525482.0679 - tn0.7: 43761380.4372 - fn0.7: 1498231.8353 - precision0.7: 0.9278 - recall0.7: 0.8191 - tp0.9: 5729492.5956 - fp0.9: 139279.5152 - tn0.9: 44147595.4887 - fn0.9: 2494276.3474 - precision0.9: 0.9
    761 - recall0.9: 0.7005 - accuracy: 0.9620 - auc: 0.9747 - val_loss: 0.9282 - val_tp0.1: 3074924.0000 - val_fp0.1: 2026526.0000 - val_tn0.1: 20054118.0000 - val_fn0.1: 1058829.0000 - val_precision0.1: 0.6028 - val_recall0.1: 0.7439 - v
    al_tp0.3: 2384857.0000 - val_fp0.3: 1512374.0000 - val_tn0.3: 20568268.0000 - val_fn0.3: 1748896.0000 - val_precision0.3: 0.6119 - val_recall0.3: 0.5769 - val_tp0.5: 1874790.0000 - val_fp0.5: 1177108.0000 - val_tn0.5: 20903548.0000 - v
    al_fn0.5: 2258963.0000 - val_precision0.5: 0.6143 - val_recall0.5: 0.4535 - val_tp0.7: 1433826.0000 - val_fp0.7: 875539.0000 - val_tn0.7: 21205108.0000 - val_fn0.7: 2699927.0000 - val_precision0.7: 0.6209 - val_recall0.7: 0.3469 - val_
    tp0.9: 876571.0000 - val_fp0.9: 463067.0000 - val_tn0.9: 21617596.0000 - val_fn0.9: 3257182.0000 - val_precision0.9: 0.6543 - val_recall0.9: 0.2121 - val_accuracy: 0.8689 - val_auc: 0.8345
    Epoch 29/50
    1280/1280 [==============================] - 3301s 3s/step - loss: 0.2631 - tp0.1: 7928634.4387 - fp0.1: 3499371.4231 - tn0.1: 40746977.2912 - fn0.1: 335655.7486 - precision0.1: 0.6931 - recall0.1: 0.9590 - tp0.3: 7590846.6565 - fp0.3:
     1745234.9133 - tn0.3: 42501135.9079 - fn0.3: 673443.5308 - precision0.3: 0.8123 - recall0.3: 0.9172 - tp0.5: 7228004.8962 - fp0.5: 987598.6276 - tn0.5: 43258775.8423 - fn0.5: 1036285.2912 - precision0.5: 0.8797 - recall0.5: 0.8722 - t
    p0.7: 6751780.5855 - fp0.7: 528906.3060 - tn0.7: 43717457.8899 - fn0.7: 1512509.6019 - precision0.7: 0.9276 - recall0.7: 0.8149 - tp0.9: 5739049.2662 - fp0.9: 134220.8212 - tn0.9: 44112121.8931 - fn0.9: 2525240.9212 - precision0.9: 0.9
    773 - recall0.9: 0.6932 - accuracy: 0.9610 - auc: 0.9735 - val_loss: 0.9733 - val_tp0.1: 2953631.0000 - val_fp0.1: 2097398.0000 - val_tn0.1: 20016580.0000 - val_fn0.1: 1146781.0000 - val_precision0.1: 0.5848 - val_recall0.1: 0.7203 - v
    al_tp0.3: 2295553.0000 - val_fp0.3: 1572391.0000 - val_tn0.3: 20541608.0000 - val_fn0.3: 1804859.0000 - val_precision0.3: 0.5935 - val_recall0.3: 0.5598 - val_tp0.5: 1907607.0000 - val_fp0.5: 1337407.0000 - val_tn0.5: 20776580.0000 - v
    al_fn0.5: 2192805.0000 - val_precision0.5: 0.5879 - val_recall0.5: 0.4652 - val_tp0.7: 1521536.0000 - val_fp0.7: 1008010.0000 - val_tn0.7: 21105980.0000 - val_fn0.7: 2578876.0000 - val_precision0.7: 0.6015 - val_recall0.7: 0.3711 - val
    _tp0.9: 972957.0000 - val_fp0.9: 483731.0000 - val_tn0.9: 21630266.0000 - val_fn0.9: 3127455.0000 - val_precision0.9: 0.6679 - val_recall0.9: 0.2373 - val_accuracy: 0.8653 - val_auc: 0.8217
    Epoch 30/50
    1280/1280 [==============================] - 3328s 3s/step - loss: 0.2476 - tp0.1: 7928433.5863 - fp0.1: 3472550.0773 - tn0.1: 40794809.0422 - fn0.1: 314855.4317 - precision0.1: 0.6960 - recall0.1: 0.9627 - tp0.3: 7601842.5277 - fp0.3:
     1687439.2631 - tn0.3: 42579912.0640 - fn0.3: 641446.4902 - precision0.3: 0.8211 - recall0.3: 0.9228 - tp0.5: 7249994.1311 - fp0.5: 954111.0289 - tn0.5: 43313245.2014 - fn0.5: 993294.8868 - precision0.5: 0.8862 - recall0.5: 0.8801 - tp
    0.7: 6788340.7728 - fp0.7: 512779.7322 - tn0.7: 43754579.6230 - fn0.7: 1454948.2451 - precision0.7: 0.9317 - recall0.7: 0.8242 - tp0.9: 5804142.8977 - fp0.9: 135882.9547 - tn0.9: 44131491.4426 - fn0.9: 2439146.1202 - precision0.9: 0.97
    77 - recall0.9: 0.7071 - accuracy: 0.9638 - auc: 0.9759 - val_loss: 0.9000 - val_tp0.1: 3155596.0000 - val_fp0.1: 2364539.0000 - val_tn0.1: 19719286.0000 - val_fn0.1: 974987.0000 - val_precision0.1: 0.5717 - val_recall0.1: 0.7640 - val
    _tp0.3: 2555024.0000 - val_fp0.3: 1820537.0000 - val_tn0.3: 20263286.0000 - val_fn0.3: 1575559.0000 - val_precision0.3: 0.5839 - val_recall0.3: 0.6186 - val_tp0.5: 2171845.0000 - val_fp0.5: 1530813.0000 - val_tn0.5: 20553012.0000 - val
    _fn0.5: 1958738.0000 - val_precision0.5: 0.5866 - val_recall0.5: 0.5258 - val_tp0.7: 1696953.0000 - val_fp0.7: 1088079.0000 - val_tn0.7: 20995736.0000 - val_fn0.7: 2433630.0000 - val_precision0.7: 0.6093 - val_recall0.7: 0.4108 - val_t
    p0.9: 986911.0000 - val_fp0.9: 596529.0000 - val_tn0.9: 21487282.0000 - val_fn0.9: 3143672.0000 - val_precision0.9: 0.6233 - val_recall0.9: 0.2389 - val_accuracy: 0.8669 - val_auc: 0.8372
    Epoch 31/50
    1280/1280 [==============================] - 3327s 3s/step - loss: 0.2474 - tp0.1: 7893463.6916 - fp0.1: 3376652.6175 - tn0.1: 40920762.5230 - fn0.1: 319751.0945 - precision0.1: 0.6988 - recall0.1: 0.9618 - tp0.3: 7584152.9766 - fp0.3:
     1687559.6604 - tn0.3: 42609861.8407 - fn0.3: 629061.8095 - precision0.3: 0.8167 - recall0.3: 0.9246 - tp0.5: 7249654.8931 - fp0.5: 952218.5824 - tn0.5: 43345227.1788 - fn0.5: 963559.8931 - precision0.5: 0.8832 - recall0.5: 0.8838 - tp
    0.7: 6813905.7268 - fp0.7: 516513.1452 - tn0.7: 43780937.7900 - fn0.7: 1399309.0593 - precision0.7: 0.9292 - recall0.7: 0.8309 - tp0.9: 5825876.9290 - fp0.9: 123739.9094 - tn0.9: 44173704.9789 - fn0.9: 2387337.8571 - precision0.9: 0.97
    94 - recall0.9: 0.7105 - accuracy: 0.9638 - auc: 0.9755 - val_loss: 1.0137 - val_tp0.1: 2843857.0000 - val_fp0.1: 2045215.0000 - val_tn0.1: 20132800.0000 - val_fn0.1: 1192532.0000 - val_precision0.1: 0.5817 - val_recall0.1: 0.7046 - va
    l_tp0.3: 2207535.0000 - val_fp0.3: 1595385.0000 - val_tn0.3: 20582624.0000 - val_fn0.3: 1828854.0000 - val_precision0.3: 0.5805 - val_recall0.3: 0.5469 - val_tp0.5: 1835971.0000 - val_fp0.5: 1390695.0000 - val_tn0.5: 20787306.0000 - va
    l_fn0.5: 2200418.0000 - val_precision0.5: 0.5690 - val_recall0.5: 0.4549 - val_tp0.7: 1407052.0000 - val_fp0.7: 1033277.0000 - val_tn0.7: 21144740.0000 - val_fn0.7: 2629337.0000 - val_precision0.7: 0.5766 - val_recall0.7: 0.3486 - val_
    tp0.9: 786994.0000 - val_fp0.9: 502079.0000 - val_tn0.9: 21675932.0000 - val_fn0.9: 3249395.0000 - val_precision0.9: 0.6105 - val_recall0.9: 0.1950 - val_accuracy: 0.8630 - val_auc: 0.8121
    Epoch 32/50
    1280/1280 [==============================] - 3314s 3s/step - loss: 0.2529 - tp0.1: 7921204.5262 - fp0.1: 3486226.5160 - tn0.1: 40787447.0554 - fn0.1: 315795.7416 - precision0.1: 0.6929 - recall0.1: 0.9624 - tp0.3: 7593409.2264 - fp0.3:
     1719995.1319 - tn0.3: 42553646.5496 - fn0.3: 643591.0414 - precision0.3: 0.8139 - recall0.3: 0.9227 - tp0.5: 7240041.4294 - fp0.5: 971849.9313 - tn0.5: 43301829.9984 - fn0.5: 996958.8384 - precision0.5: 0.8807 - recall0.5: 0.8794 - tp
    0.7: 6782718.2459 - fp0.7: 527577.0203 - tn0.7: 43746081.8454 - fn0.7: 1454282.0219 - precision0.7: 0.9273 - recall0.7: 0.8235 - tp0.9: 5780993.2069 - fp0.9: 137794.4926 - tn0.9: 44135860.1952 - fn0.9: 2456007.0609 - precision0.9: 0.97
    62 - recall0.9: 0.7029 - accuracy: 0.9623 - auc: 0.9755 - val_loss: 0.9139 - val_tp0.1: 2984184.0000 - val_fp0.1: 1853144.0000 - val_tn0.1: 20302900.0000 - val_fn0.1: 1074177.0000 - val_precision0.1: 0.6169 - val_recall0.1: 0.7353 - va
    l_tp0.3: 2282389.0000 - val_fp0.3: 1253127.0000 - val_tn0.3: 20902908.0000 - val_fn0.3: 1775972.0000 - val_precision0.3: 0.6456 - val_recall0.3: 0.5624 - val_tp0.5: 1856088.0000 - val_fp0.5: 1008268.0000 - val_tn0.5: 21147768.0000 - va
    l_fn0.5: 2202273.0000 - val_precision0.5: 0.6480 - val_recall0.5: 0.4573 - val_tp0.7: 1412568.0000 - val_fp0.7: 746394.0000 - val_tn0.7: 21409650.0000 - val_fn0.7: 2645793.0000 - val_precision0.7: 0.6543 - val_recall0.7: 0.3481 - val_t
    p0.9: 859420.0000 - val_fp0.9: 382687.0000 - val_tn0.9: 21773356.0000 - val_fn0.9: 3198941.0000 - val_precision0.9: 0.6919 - val_recall0.9: 0.2118 - val_accuracy: 0.8775 - val_auc: 0.8374
    Epoch 33/50
    1280/1280 [==============================] - 3327s 3s/step - loss: 0.2503 - tp0.1: 8047970.1561 - fp0.1: 3391824.5386 - tn0.1: 40757551.3286 - fn0.1: 313307.7994 - precision0.1: 0.7045 - recall0.1: 0.9628 - tp0.3: 7729505.2756 - fp0.3:
     1696243.9040 - tn0.3: 42453121.2498 - fn0.3: 631772.6799 - precision0.3: 0.8217 - recall0.3: 0.9244 - tp0.5: 7388192.9508 - fp0.5: 982075.0718 - tn0.5: 43167288.1382 - fn0.5: 973085.0047 - precision0.5: 0.8838 - recall0.5: 0.8830 - tp
    0.7: 6931423.2888 - fp0.7: 537102.3271 - tn0.7: 43612270.6479 - fn0.7: 1429854.6667 - precision0.7: 0.9287 - recall0.7: 0.8280 - tp0.9: 5907919.1241 - fp0.9: 136823.9852 - tn0.9: 44012543.5980 - fn0.9: 2453358.8314 - precision0.9: 0.97
    77 - recall0.9: 0.7060 - accuracy: 0.9628 - auc: 0.9760 - val_loss: 1.0074 - val_tp0.1: 3175440.0000 - val_fp0.1: 2183311.0000 - val_tn0.1: 19707500.0000 - val_fn0.1: 1148150.0000 - val_precision0.1: 0.5926 - val_recall0.1: 0.7344 - va
    l_tp0.3: 2561350.0000 - val_fp0.3: 1772975.0000 - val_tn0.3: 20117834.0000 - val_fn0.3: 1762240.0000 - val_precision0.3: 0.5909 - val_recall0.3: 0.5924 - val_tp0.5: 2236272.0000 - val_fp0.5: 1567765.0000 - val_tn0.5: 20323044.0000 - va
    l_fn0.5: 2087318.0000 - val_precision0.5: 0.5879 - val_recall0.5: 0.5172 - val_tp0.7: 1860628.0000 - val_fp0.7: 1235466.0000 - val_tn0.7: 20655340.0000 - val_fn0.7: 2462962.0000 - val_precision0.7: 0.6010 - val_recall0.7: 0.4303 - val_
    tp0.9: 1296792.0000 - val_fp0.9: 704030.0000 - val_tn0.9: 21186792.0000 - val_fn0.9: 3026798.0000 - val_precision0.9: 0.6481 - val_recall0.9: 0.2999 - val_accuracy: 0.8606 - val_auc: 0.8254
    Epoch 34/50
    1280/1280 [==============================] - 3324s 3s/step - loss: 0.2507 - tp0.1: 7911580.4098 - fp0.1: 3345203.1913 - tn0.1: 40931131.7447 - fn0.1: 322742.2100 - precision0.1: 0.7028 - recall0.1: 0.9609 - tp0.3: 7599881.2022 - fp0.3:
     1647150.7923 - tn0.3: 42629169.7291 - fn0.3: 634441.4176 - precision0.3: 0.8220 - recall0.3: 0.9223 - tp0.5: 7270919.3005 - fp0.5: 951517.5379 - tn0.5: 43324815.6550 - fn0.5: 963403.3193 - precision0.5: 0.8843 - recall0.5: 0.8819 - tp
    0.7: 6834573.1491 - fp0.7: 523079.1390 - tn0.7: 43753256.2943 - fn0.7: 1399749.4707 - precision0.7: 0.9292 - recall0.7: 0.8288 - tp0.9: 5841839.3435 - fp0.9: 132340.8064 - tn0.9: 44144004.1702 - fn0.9: 2392483.2763 - precision0.9: 0.97
    80 - recall0.9: 0.7081 - accuracy: 0.9633 - auc: 0.9751 - val_loss: 0.9237 - val_tp0.1: 3263836.0000 - val_fp0.1: 1919441.0000 - val_tn0.1: 19948316.0000 - val_fn0.1: 1082810.0000 - val_precision0.1: 0.6297 - val_recall0.1: 0.7509 - va
    l_tp0.3: 2560242.0000 - val_fp0.3: 1398314.0000 - val_tn0.3: 20469446.0000 - val_fn0.3: 1786404.0000 - val_precision0.3: 0.6468 - val_recall0.3: 0.5890 - val_tp0.5: 2148456.0000 - val_fp0.5: 1146771.0000 - val_tn0.5: 20720984.0000 - va
    l_fn0.5: 2198190.0000 - val_precision0.5: 0.6520 - val_recall0.5: 0.4943 - val_tp0.7: 1663813.0000 - val_fp0.7: 806718.0000 - val_tn0.7: 21061036.0000 - val_fn0.7: 2682833.0000 - val_precision0.7: 0.6735 - val_recall0.7: 0.3828 - val_t
    p0.9: 979165.0000 - val_fp0.9: 394579.0000 - val_tn0.9: 21473184.0000 - val_fn0.9: 3367481.0000 - val_precision0.9: 0.7128 - val_recall0.9: 0.2253 - val_accuracy: 0.8724 - val_auc: 0.8405
    Epoch 35/50
    1280/1280 [==============================] - 3310s 3s/step - loss: 0.2614 - tp0.1: 7937448.7291 - fp0.1: 3421955.0578 - tn0.1: 40822462.2030 - fn0.1: 328785.1717 - precision0.1: 0.6999 - recall0.1: 0.9579 - tp0.3: 7621030.6534 - fp0.3:
     1703065.0414 - tn0.3: 42541374.6472 - fn0.3: 645203.2475 - precision0.3: 0.8174 - recall0.3: 0.9200 - tp0.5: 7279229.2404 - fp0.5: 977595.1093 - tn0.5: 43266832.8696 - fn0.5: 987004.6604 - precision0.5: 0.8809 - recall0.5: 0.8792 - tp
    0.7: 6838107.3216 - fp0.7: 536670.7908 - tn0.7: 43707710.4473 - fn0.7: 1428126.5792 - precision0.7: 0.9261 - recall0.7: 0.8262 - tp0.9: 5811902.2927 - fp0.9: 135171.2490 - tn0.9: 44109247.1507 - fn0.9: 2454331.6081 - precision0.9: 0.97
    71 - recall0.9: 0.7018 - accuracy: 0.9620 - auc: 0.9732 - val_loss: 0.8013 - val_tp0.1: 3284754.0000 - val_fp0.1: 2344554.0000 - val_tn0.1: 19771954.0000 - val_fn0.1: 813139.0000 - val_precision0.1: 0.5835 - val_recall0.1: 0.8016 - val
    _tp0.3: 2575560.0000 - val_fp0.3: 1777769.0000 - val_tn0.3: 20338744.0000 - val_fn0.3: 1522333.0000 - val_precision0.3: 0.5916 - val_recall0.3: 0.6285 - val_tp0.5: 2179640.0000 - val_fp0.5: 1496577.0000 - val_tn0.5: 20619926.0000 - val
    _fn0.5: 1918253.0000 - val_precision0.5: 0.5929 - val_recall0.5: 0.5319 - val_tp0.7: 1701082.0000 - val_fp0.7: 1053012.0000 - val_tn0.7: 21063498.0000 - val_fn0.7: 2396811.0000 - val_precision0.7: 0.6177 - val_recall0.7: 0.4151 - val_t
    p0.9: 989696.0000 - val_fp0.9: 527144.0000 - val_tn0.9: 21589364.0000 - val_fn0.9: 3108197.0000 - val_precision0.9: 0.6525 - val_recall0.9: 0.2415 - val_accuracy: 0.8697 - val_auc: 0.8572
    Epoch 36/50
    1280/1280 [==============================] - 3341s 3s/step - loss: 0.2585 - tp0.1: 7873502.4785 - fp0.1: 3351134.0312 - tn0.1: 40955766.1046 - fn0.1: 330264.3185 - precision0.1: 0.6971 - recall0.1: 0.9588 - tp0.3: 7549771.1140 - fp0.3:
     1661219.6745 - tn0.3: 42645646.7143 - fn0.3: 653995.6831 - precision0.3: 0.8162 - recall0.3: 0.9183 - tp0.5: 7211353.4215 - fp0.5: 964176.2467 - tn0.5: 43342703.0515 - fn0.5: 992413.3755 - precision0.5: 0.8793 - recall0.5: 0.8765 - tp
    0.7: 6769812.5121 - fp0.7: 536905.1553 - tn0.7: 43770008.0851 - fn0.7: 1433954.2849 - precision0.7: 0.9245 - recall0.7: 0.8228 - tp0.9: 5783984.8657 - fp0.9: 143730.5753 - tn0.9: 44163162.1390 - fn0.9: 2419781.9313 - precision0.9: 0.97
    52 - recall0.9: 0.7042 - accuracy: 0.9619 - auc: 0.9736 - val_loss: 0.7459 - val_tp0.1: 3472629.0000 - val_fp0.1: 2432774.0000 - val_tn0.1: 19591824.0000 - val_fn0.1: 717165.0000 - val_precision0.1: 0.5880 - val_recall0.1: 0.8288 - val
    _tp0.3: 2809063.0000 - val_fp0.3: 1835029.0000 - val_tn0.3: 20189584.0000 - val_fn0.3: 1380731.0000 - val_precision0.3: 0.6049 - val_recall0.3: 0.6705 - val_tp0.5: 2315959.0000 - val_fp0.5: 1490932.0000 - val_tn0.5: 20533678.0000 - val
    _fn0.5: 1873835.0000 - val_precision0.5: 0.6084 - val_recall0.5: 0.5528 - val_tp0.7: 1776363.0000 - val_fp0.7: 1094378.0000 - val_tn0.7: 20930232.0000 - val_fn0.7: 2413431.0000 - val_precision0.7: 0.6188 - val_recall0.7: 0.4240 - val_t
    p0.9: 1035383.0000 - val_fp0.9: 541763.0000 - val_tn0.9: 21482840.0000 - val_fn0.9: 3154411.0000 - val_precision0.9: 0.6565 - val_recall0.9: 0.2471 - val_accuracy: 0.8716 - val_auc: 0.8687
    Epoch 37/50
    1280/1280 [==============================] - 3295s 3s/step - loss: 0.2437 - tp0.1: 8016183.8337 - fp0.1: 3307100.4614 - tn0.1: 40879834.5652 - fn0.1: 307556.7455 - precision0.1: 0.7070 - recall0.1: 0.9629 - tp0.3: 7712574.3755 - fp0.3:
     1659818.0539 - tn0.3: 42527102.4731 - fn0.3: 611166.2037 - precision0.3: 0.8224 - recall0.3: 0.9270 - tp0.5: 7388699.9188 - fp0.5: 964567.5652 - tn0.5: 43222331.7237 - fn0.5: 935040.6604 - precision0.5: 0.8843 - recall0.5: 0.8884 - tp
    0.7: 6943361.7510 - fp0.7: 525733.5105 - tn0.7: 43661155.2553 - fn0.7: 1380378.8283 - precision0.7: 0.9294 - recall0.7: 0.8349 - tp0.9: 5919522.7244 - fp0.9: 126122.8407 - tn0.9: 44060797.3005 - fn0.9: 2404217.8548 - precision0.9: 0.97
    90 - recall0.9: 0.7122 - accuracy: 0.9641 - auc: 0.9762 - val_loss: 0.8760 - val_tp0.1: 3352442.0000 - val_fp0.1: 2392549.0000 - val_tn0.1: 19544092.0000 - val_fn0.1: 925309.0000 - val_precision0.1: 0.5835 - val_recall0.1: 0.7837 - val
    _tp0.3: 2658571.0000 - val_fp0.3: 1872790.0000 - val_tn0.3: 20063866.0000 - val_fn0.3: 1619180.0000 - val_precision0.3: 0.5867 - val_recall0.3: 0.6215 - val_tp0.5: 2313864.0000 - val_fp0.5: 1697140.0000 - val_tn0.5: 20239508.0000 - val
    _fn0.5: 1963887.0000 - val_precision0.5: 0.5769 - val_recall0.5: 0.5409 - val_tp0.7: 1882798.0000 - val_fp0.7: 1331096.0000 - val_tn0.7: 20605548.0000 - val_fn0.7: 2394953.0000 - val_precision0.7: 0.5858 - val_recall0.7: 0.4401 - val_t
    p0.9: 1104516.0000 - val_fp0.9: 586104.0000 - val_tn0.9: 21350548.0000 - val_fn0.9: 3173235.0000 - val_precision0.9: 0.6533 - val_recall0.9: 0.2582 - val_accuracy: 0.8603 - val_auc: 0.8446
    Epoch 38/50
    1280/1280 [==============================] - 3369s 3s/step - loss: 0.2413 - tp0.1: 7956859.0180 - fp0.1: 3276078.4895 - tn0.1: 40979600.7861 - fn0.1: 298124.8735 - precision0.1: 0.7090 - recall0.1: 0.9634 - tp0.3: 7646937.8517 - fp0.3:
     1590794.5269 - tn0.3: 42664873.7775 - fn0.3: 608046.0398 - precision0.3: 0.8280 - recall0.3: 0.9258 - tp0.5: 7313832.8025 - fp0.5: 911027.5652 - tn0.5: 43344624.5683 - fn0.5: 941151.0890 - precision0.5: 0.8894 - recall0.5: 0.8851 - tp
    0.7: 6881091.8657 - fp0.7: 495880.5730 - tn0.7: 43759809.8337 - fn0.7: 1373892.0258 - precision0.7: 0.9331 - recall0.7: 0.8326 - tp0.9: 5927602.7518 - fp0.9: 127784.9633 - tn0.9: 44127895.0578 - fn0.9: 2327381.1397 - precision0.9: 0.97
    90 - recall0.9: 0.7159 - accuracy: 0.9644 - auc: 0.9765 - val_loss: 4.9077 - val_tp0.1: 3486973.0000 - val_fp0.1: 2850284.0000 - val_tn0.1: 19285722.0000 - val_fn0.1: 591422.0000 - val_precision0.1: 0.5502 - val_recall0.1: 0.8550 - val
    _tp0.3: 2787062.0000 - val_fp0.3: 2180072.0000 - val_tn0.3: 19955928.0000 - val_fn0.3: 1291333.0000 - val_precision0.3: 0.5611 - val_recall0.3: 0.6834 - val_tp0.5: 2371435.0000 - val_fp0.5: 2041150.0000 - val_tn0.5: 20094856.0000 - val
    _fn0.5: 1706960.0000 - val_precision0.5: 0.5374 - val_recall0.5: 0.5815 - val_tp0.7: 2052507.0000 - val_fp0.7: 1911080.0000 - val_tn0.7: 20224928.0000 - val_fn0.7: 2025888.0000 - val_precision0.7: 0.5178 - val_recall0.7: 0.5033 - val_t
    p0.9: 1593600.0000 - val_fp0.9: 1598028.0000 - val_tn0.9: 20537976.0000 - val_fn0.9: 2484795.0000 - val_precision0.9: 0.4993 - val_recall0.9: 0.3907 - val_accuracy: 0.8570 - val_auc: 0.8622
    Epoch 39/50
    1280/1280 [==============================] - 3380s 3s/step - loss: 0.2414 - tp0.1: 7803821.4372 - fp0.1: 3354269.5839 - tn0.1: 41052380.7252 - fn0.1: 300171.4941 - precision0.1: 0.6956 - recall0.1: 0.9633 - tp0.3: 7497167.4637 - fp0.3:
     1643206.8829 - tn0.3: 42763432.2763 - fn0.3: 606825.4676 - precision0.3: 0.8198 - recall0.3: 0.9250 - tp0.5: 7161284.4465 - fp0.5: 930784.7471 - tn0.5: 43475886.7112 - fn0.5: 942708.4848 - precision0.5: 0.8867 - recall0.5: 0.8828 - tp
    0.7: 6716037.1132 - fp0.7: 498289.9742 - tn0.7: 43908382.7361 - fn0.7: 1387955.8181 - precision0.7: 0.9325 - recall0.7: 0.8275 - tp0.9: 5745482.9321 - fp0.9: 129539.1460 - tn0.9: 44277093.8798 - fn0.9: 2358509.9992 - precision0.9: 0.97
    87 - recall0.9: 0.7074 - accuracy: 0.9647 - auc: 0.9763 - val_loss: 0.8416 - val_tp0.1: 3152764.0000 - val_fp0.1: 1979016.0000 - val_tn0.1: 20133118.0000 - val_fn0.1: 949499.0000 - val_precision0.1: 0.6144 - val_recall0.1: 0.7685 - val
    _tp0.3: 2438228.0000 - val_fp0.3: 1470054.0000 - val_tn0.3: 20642072.0000 - val_fn0.3: 1664035.0000 - val_precision0.3: 0.6239 - val_recall0.3: 0.5944 - val_tp0.5: 1977683.0000 - val_fp0.5: 1217609.0000 - val_tn0.5: 20894528.0000 - val
    _fn0.5: 2124580.0000 - val_precision0.5: 0.6189 - val_recall0.5: 0.4821 - val_tp0.7: 1503343.0000 - val_fp0.7: 862897.0000 - val_tn0.7: 21249240.0000 - val_fn0.7: 2598920.0000 - val_precision0.7: 0.6353 - val_recall0.7: 0.3665 - val_tp
    0.9: 896812.0000 - val_fp0.9: 377234.0000 - val_tn0.9: 21734904.0000 - val_fn0.9: 3205451.0000 - val_precision0.9: 0.7039 - val_recall0.9: 0.2186 - val_accuracy: 0.8725 - val_auc: 0.8477
    Epoch 40/50
    1280/1280 [==============================] - 3386s 3s/step - loss: 0.2375 - tp0.1: 8019265.8907 - fp0.1: 3238880.4543 - tn0.1: 40953284.0133 - fn0.1: 299240.1600 - precision0.1: 0.7138 - recall0.1: 0.9637 - tp0.3: 7719720.4668 - fp0.3:
     1605122.1085 - tn0.3: 42587020.9703 - fn0.3: 598785.5839 - precision0.3: 0.8287 - recall0.3: 0.9280 - tp0.5: 7396785.5230 - fp0.5: 927286.7697 - tn0.5: 43264845.8103 - fn0.5: 921720.5277 - precision0.5: 0.8890 - recall0.5: 0.8895 - tp
    0.7: 6965247.9297 - fp0.7: 501804.8829 - tn0.7: 43690323.6995 - fn0.7: 1353258.1210 - precision0.7: 0.9328 - recall0.7: 0.8383 - tp0.9: 5960658.1421 - fp0.9: 123056.3981 - tn0.9: 44069097.9461 - fn0.9: 2357847.9087 - precision0.9: 0.97
    98 - recall0.9: 0.7181 - accuracy: 0.9649 - auc: 0.9769 - val_loss: 0.7336 - val_tp0.1: 3398410.0000 - val_fp0.1: 2457064.0000 - val_tn0.1: 19690942.0000 - val_fn0.1: 667979.0000 - val_precision0.1: 0.5804 - val_recall0.1: 0.8357 - val
    _tp0.3: 2749737.0000 - val_fp0.3: 1843797.0000 - val_tn0.3: 20304204.0000 - val_fn0.3: 1316652.0000 - val_precision0.3: 0.5986 - val_recall0.3: 0.6762 - val_tp0.5: 2345968.0000 - val_fp0.5: 1609916.0000 - val_tn0.5: 20538100.0000 - val
    _fn0.5: 1720421.0000 - val_precision0.5: 0.5930 - val_recall0.5: 0.5769 - val_tp0.7: 1892495.0000 - val_fp0.7: 1225411.0000 - val_tn0.7: 20922604.0000 - val_fn0.7: 2173894.0000 - val_precision0.7: 0.6070 - val_recall0.7: 0.4654 - val_t
    p0.9: 1133144.0000 - val_fp0.9: 662854.0000 - val_tn0.9: 21485156.0000 - val_fn0.9: 2933245.0000 - val_precision0.9: 0.6309 - val_recall0.9: 0.2787 - val_accuracy: 0.8730 - val_auc: 0.8721
    Epoch 41/50
    1280/1280 [==============================] - 3385s 3s/step - loss: 0.2342 - tp0.1: 7776741.3099 - fp0.1: 3289425.7650 - tn0.1: 41160217.6448 - fn0.1: 284258.4153 - precision0.1: 0.7028 - recall0.1: 0.9655 - tp0.3: 7475615.4660 - fp0.3:
     1607131.3685 - tn0.3: 42842547.8212 - fn0.3: 585384.2592 - precision0.3: 0.8230 - recall0.3: 0.9281 - tp0.5: 7146650.6237 - fp0.5: 908662.7939 - tn0.5: 43541000.6799 - fn0.5: 914349.1015 - precision0.5: 0.8871 - recall0.5: 0.8872 - tp
    0.7: 6711653.7861 - fp0.7: 487381.3060 - tn0.7: 43962279.2373 - fn0.7: 1349345.9391 - precision0.7: 0.9322 - recall0.7: 0.8330 - tp0.9: 5745993.2186 - fp0.9: 124517.9758 - tn0.9: 44325130.4988 - fn0.9: 2315006.5066 - precision0.9: 0.97
    85 - recall0.9: 0.7118 - accuracy: 0.9651 - auc: 0.9776 - val_loss: 0.9870 - val_tp0.1: 3053092.0000 - val_fp0.1: 1964937.0000 - val_tn0.1: 20015960.0000 - val_fn0.1: 1180406.0000 - val_precision0.1: 0.6084 - val_recall0.1: 0.7212 - va
    l_tp0.3: 2284036.0000 - val_fp0.3: 1477236.0000 - val_tn0.3: 20503676.0000 - val_fn0.3: 1949462.0000 - val_precision0.3: 0.6073 - val_recall0.3: 0.5395 - val_tp0.5: 1899487.0000 - val_fp0.5: 1256383.0000 - val_tn0.5: 20724524.0000 - va
    l_fn0.5: 2334011.0000 - val_precision0.5: 0.6019 - val_recall0.5: 0.4487 - val_tp0.7: 1422839.0000 - val_fp0.7: 921281.0000 - val_tn0.7: 21059624.0000 - val_fn0.7: 2810659.0000 - val_precision0.7: 0.6070 - val_recall0.7: 0.3361 - val_t
    p0.9: 829426.0000 - val_fp0.9: 481727.0000 - val_tn0.9: 21499180.0000 - val_fn0.9: 3404072.0000 - val_precision0.9: 0.6326 - val_recall0.9: 0.1959 - val_accuracy: 0.8630 - val_auc: 0.8226
    Epoch 42/50
    1280/1280 [==============================] - 3315s 3s/step - loss: 0.2346 - tp0.1: 8105380.3060 - fp0.1: 3217998.5480 - tn0.1: 40897704.3310 - fn0.1: 289593.6690 - precision0.1: 0.7181 - recall0.1: 0.9656 - tp0.3: 7802382.4910 - fp0.3:
     1604270.8267 - tn0.3: 42511413.2233 - fn0.3: 592591.4840 - precision0.3: 0.8318 - recall0.3: 0.9298 - tp0.5: 7475287.1304 - fp0.5: 928889.2771 - tn0.5: 43186785.4551 - fn0.5: 919686.8447 - precision0.5: 0.8906 - recall0.5: 0.8917 - tp
    0.7: 7046455.8064 - fp0.7: 507457.0898 - tn0.7: 43608208.9945 - fn0.7: 1348518.1686 - precision0.7: 0.9334 - recall0.7: 0.8413 - tp0.9: 6058878.7190 - fp0.9: 128713.2311 - tn0.9: 43986958.0468 - fn0.9: 2336095.2560 - precision0.9: 0.97
    95 - recall0.9: 0.7240 - accuracy: 0.9650 - auc: 0.9779 - val_loss: 0.7327 - val_tp0.1: 3402300.0000 - val_fp0.1: 2653288.0000 - val_tn0.1: 19553712.0000 - val_fn0.1: 605094.0000 - val_precision0.1: 0.5618 - val_recall0.1: 0.8490 - val
    _tp0.3: 2812815.0000 - val_fp0.3: 2101002.0000 - val_tn0.3: 20106008.0000 - val_fn0.3: 1194579.0000 - val_precision0.3: 0.5724 - val_recall0.3: 0.7019 - val_tp0.5: 2455006.0000 - val_fp0.5: 1878473.0000 - val_tn0.5: 20328530.0000 - val
    _fn0.5: 1552388.0000 - val_precision0.5: 0.5665 - val_recall0.5: 0.6126 - val_tp0.7: 2074881.0000 - val_fp0.7: 1530207.0000 - val_tn0.7: 20676798.0000 - val_fn0.7: 1932513.0000 - val_precision0.7: 0.5755 - val_recall0.7: 0.5178 - val_t
    p0.9: 1484583.0000 - val_fp0.9: 913417.0000 - val_tn0.9: 21293592.0000 - val_fn0.9: 2522811.0000 - val_precision0.9: 0.6191 - val_recall0.9: 0.3705 - val_accuracy: 0.8691 - val_auc: 0.8751
    Epoch 43/50
    1280/1280 [==============================] - 3323s 3s/step - loss: 0.2312 - tp0.1: 7991710.0656 - fp0.1: 3249384.0929 - tn0.1: 40992411.0070 - fn0.1: 277145.7760 - precision0.1: 0.7097 - recall0.1: 0.9664 - tp0.3: 7693901.7158 - fp0.3:
     1595395.2670 - tn0.3: 42646409.5269 - fn0.3: 574954.1257 - precision0.3: 0.8268 - recall0.3: 0.9305 - tp0.5: 7373192.4247 - fp0.5: 917265.3872 - tn0.5: 43324540.4536 - fn0.5: 895663.4169 - precision0.5: 0.8882 - recall0.5: 0.8913 - tp
    0.7: 6950466.7244 - fp0.7: 497989.1148 - tn0.7: 43743813.0945 - fn0.7: 1318389.1171 - precision0.7: 0.9326 - recall0.7: 0.8395 - tp0.9: 5981119.6440 - fp0.9: 128545.2209 - tn0.9: 44113239.2467 - fn0.9: 2287736.1975 - precision0.9: 0.97
    92 - recall0.9: 0.7218 - accuracy: 0.9653 - auc: 0.9783 - val_loss: 0.9647 - val_tp0.1: 2876343.0000 - val_fp0.1: 1660425.0000 - val_tn0.1: 20507034.0000 - val_fn0.1: 1170598.0000 - val_precision0.1: 0.6340 - val_recall0.1: 0.7107 - va
    l_tp0.3: 2209087.0000 - val_fp0.3: 1251945.0000 - val_tn0.3: 20915512.0000 - val_fn0.3: 1837854.0000 - val_precision0.3: 0.6383 - val_recall0.3: 0.5459 - val_tp0.5: 1778622.0000 - val_fp0.5: 995657.0000 - val_tn0.5: 21171812.0000 - val
    _fn0.5: 2268319.0000 - val_precision0.5: 0.6411 - val_recall0.5: 0.4395 - val_tp0.7: 1307626.0000 - val_fp0.7: 702184.0000 - val_tn0.7: 21465282.0000 - val_fn0.7: 2739315.0000 - val_precision0.7: 0.6506 - val_recall0.7: 0.3231 - val_tp
    0.9: 721015.0000 - val_fp0.9: 304685.0000 - val_tn0.9: 21862786.0000 - val_fn0.9: 3325926.0000 - val_precision0.9: 0.7029 - val_recall0.9: 0.1782 - val_accuracy: 0.8755 - val_auc: 0.8252
    Epoch 44/50
    1280/1280 [==============================] - 3324s 3s/step - loss: 0.2316 - tp0.1: 7993553.5519 - fp0.1: 3116269.6425 - tn0.1: 41108883.5878 - fn0.1: 291939.1897 - precision0.1: 0.7205 - recall0.1: 0.9644 - tp0.3: 7697513.5496 - fp0.3:
     1532368.8548 - tn0.3: 42692802.8033 - fn0.3: 587979.1920 - precision0.3: 0.8347 - recall0.3: 0.9284 - tp0.5: 7391393.2475 - fp0.5: 891003.0500 - tn0.5: 43334168.9118 - fn0.5: 894099.4941 - precision0.5: 0.8927 - recall0.5: 0.8916 - tp
    0.7: 6972420.4403 - fp0.7: 485629.1991 - tn0.7: 43739548.5152 - fn0.7: 1313072.3013 - precision0.7: 0.9351 - recall0.7: 0.8406 - tp0.9: 6005577.5371 - fp0.9: 125956.1194 - tn0.9: 44099185.2194 - fn0.9: 2279915.2045 - precision0.9: 0.97
    93 - recall0.9: 0.7244 - accuracy: 0.9660 - auc: 0.9775 - val_loss: 0.7395 - val_tp0.1: 3516454.0000 - val_fp0.1: 2931698.0000 - val_tn0.1: 19145124.0000 - val_fn0.1: 621132.0000 - val_precision0.1: 0.5453 - val_recall0.1: 0.8499 - val
    _tp0.3: 2802624.0000 - val_fp0.3: 2096050.0000 - val_tn0.3: 19980764.0000 - val_fn0.3: 1334962.0000 - val_precision0.3: 0.5721 - val_recall0.3: 0.6774 - val_tp0.5: 2395920.0000 - val_fp0.5: 1817727.0000 - val_tn0.5: 20259092.0000 - val
    _fn0.5: 1741666.0000 - val_precision0.5: 0.5686 - val_recall0.5: 0.5791 - val_tp0.7: 1884999.0000 - val_fp0.7: 1378065.0000 - val_tn0.7: 20698756.0000 - val_fn0.7: 2252587.0000 - val_precision0.7: 0.5777 - val_recall0.7: 0.4556 - val_t
    p0.9: 983346.0000 - val_fp0.9: 679335.0000 - val_tn0.9: 21397474.0000 - val_fn0.9: 3154240.0000 - val_precision0.9: 0.5914 - val_recall0.9: 0.2377 - val_accuracy: 0.8642 - val_auc: 0.8683
    Epoch 45/50
    1280/1280 [==============================] - 3312s 3s/step - loss: 0.2330 - tp0.1: 7970480.2670 - fp0.1: 3209589.4871 - tn0.1: 41051845.7307 - fn0.1: 278741.1163 - precision0.1: 0.7127 - recall0.1: 0.9666 - tp0.3: 7675031.2553 - fp0.3:
     1589895.0429 - tn0.3: 42671527.8275 - fn0.3: 574190.1280 - precision0.3: 0.8279 - recall0.3: 0.9311 - tp0.5: 7351503.7557 - fp0.5: 918595.6940 - tn0.5: 43342860.1577 - fn0.5: 897717.6276 - precision0.5: 0.8886 - recall0.5: 0.8921 - tp
    0.7: 6925182.0765 - fp0.7: 503433.8673 - tn0.7: 43758011.1593 - fn0.7: 1324039.3068 - precision0.7: 0.9318 - recall0.7: 0.8413 - tp0.9: 5948949.5558 - fp0.9: 139165.2740 - tn0.9: 44122272.1593 - fn0.9: 2300271.8275 - precision0.9: 0.97
    68 - recall0.9: 0.7246 - accuracy: 0.9651 - auc: 0.9781 - val_loss: 0.7183 - val_tp0.1: 3462944.0000 - val_fp0.1: 2547992.0000 - val_tn0.1: 19584976.0000 - val_fn0.1: 618487.0000 - val_precision0.1: 0.5761 - val_recall0.1: 0.8485 - val
    _tp0.3: 2757155.0000 - val_fp0.3: 1916541.0000 - val_tn0.3: 20216436.0000 - val_fn0.3: 1324276.0000 - val_precision0.3: 0.5899 - val_recall0.3: 0.6755 - val_tp0.5: 2406562.0000 - val_fp0.5: 1766897.0000 - val_tn0.5: 20366082.0000 - val
    _fn0.5: 1674869.0000 - val_precision0.5: 0.5766 - val_recall0.5: 0.5896 - val_tp0.7: 2083375.0000 - val_fp0.7: 1557338.0000 - val_tn0.7: 20575626.0000 - val_fn0.7: 1998056.0000 - val_precision0.7: 0.5722 - val_recall0.7: 0.5105 - val_t
    p0.9: 1330037.0000 - val_fp0.9: 696335.0000 - val_tn0.9: 21436634.0000 - val_fn0.9: 2751394.0000 - val_precision0.9: 0.6564 - val_recall0.9: 0.3259 - val_accuracy: 0.8687 - val_auc: 0.8762
    Epoch 46/50
    1280/1280 [==============================] - 3307s 3s/step - loss: 0.2304 - tp0.1: 7916214.3216 - fp0.1: 3181145.9766 - tn0.1: 41127786.5941 - fn0.1: 285496.9227 - precision0.1: 0.7125 - recall0.1: 0.9651 - tp0.3: 7621058.9899 - fp0.3:
     1584443.6526 - tn0.3: 42724483.9516 - fn0.3: 580652.2545 - precision0.3: 0.8279 - recall0.3: 0.9291 - tp0.5: 7291826.7104 - fp0.5: 896453.6901 - tn0.5: 43412477.0523 - fn0.5: 909884.5340 - precision0.5: 0.8906 - recall0.5: 0.8889 - tp
    0.7: 6852507.8103 - fp0.7: 474238.7486 - tn0.7: 43834734.6058 - fn0.7: 1349203.4340 - precision0.7: 0.9355 - recall0.7: 0.8352 - tp0.9: 5907297.4052 - fp0.9: 120870.0991 - tn0.9: 44188071.7377 - fn0.9: 2294413.8392 - precision0.9: 0.98
    03 - recall0.9: 0.7195 - accuracy: 0.9658 - auc: 0.9776 - val_loss: 0.8308 - val_tp0.1: 2925415.0000 - val_fp0.1: 999631.0000 - val_tn0.1: 21181596.0000 - val_fn0.1: 1107748.0000 - val_precision0.1: 0.7453 - val_recall0.1: 0.7253 - val
    _tp0.3: 2129554.0000 - val_fp0.3: 432959.0000 - val_tn0.3: 21748276.0000 - val_fn0.3: 1903609.0000 - val_precision0.3: 0.8310 - val_recall0.3: 0.5280 - val_tp0.5: 1616477.0000 - val_fp0.5: 255434.0000 - val_tn0.5: 21925804.0000 - val_f
    n0.5: 2416686.0000 - val_precision0.5: 0.8635 - val_recall0.5: 0.4008 - val_tp0.7: 1160120.0000 - val_fp0.7: 159480.0000 - val_tn0.7: 22021744.0000 - val_fn0.7: 2873043.0000 - val_precision0.7: 0.8791 - val_recall0.7: 0.2876 - val_tp0.
    9: 660411.0000 - val_fp0.9: 58454.0000 - val_tn0.9: 22122774.0000 - val_fn0.9: 3372752.0000 - val_precision0.9: 0.9187 - val_recall0.9: 0.1637 - val_accuracy: 0.8981 - val_auc: 0.8530
    Epoch 47/50
    1280/1280 [==============================] - 3373s 3s/step - loss: 0.2224 - tp0.1: 7905453.9578 - fp0.1: 3088525.5402 - tn0.1: 41239482.1889 - fn0.1: 277215.0070 - precision0.1: 0.7202 - recall0.1: 0.9667 - tp0.3: 7625498.5870 - fp0.3:
     1552664.1967 - tn0.3: 42775315.1663 - fn0.3: 557170.3778 - precision0.3: 0.8312 - recall0.3: 0.9328 - tp0.5: 7314136.2771 - fp0.5: 889383.1952 - tn0.5: 43438616.4731 - fn0.5: 868532.6877 - precision0.5: 0.8918 - recall0.5: 0.8950 - tp
    0.7: 6893069.8415 - fp0.7: 475384.9407 - tn0.7: 43852616.7221 - fn0.7: 1289599.1233 - precision0.7: 0.9360 - recall0.7: 0.8437 - tp0.9: 5954104.8665 - fp0.9: 121730.6745 - tn0.9: 44206239.8525 - fn0.9: 2228564.0984 - precision0.9: 0.98
    04 - recall0.9: 0.7290 - accuracy: 0.9667 - auc: 0.9788 - val_loss: 1.0397 - val_tp0.1: 2921669.0000 - val_fp0.1: 1566454.0000 - val_tn0.1: 20378636.0000 - val_fn0.1: 1347646.0000 - val_precision0.1: 0.6510 - val_recall0.1: 0.6843 - va
    l_tp0.3: 2142218.0000 - val_fp0.3: 1055434.0000 - val_tn0.3: 20889652.0000 - val_fn0.3: 2127097.0000 - val_precision0.3: 0.6699 - val_recall0.3: 0.5018 - val_tp0.5: 1695622.0000 - val_fp0.5: 805323.0000 - val_tn0.5: 21139764.0000 - val
    _fn0.5: 2573693.0000 - val_precision0.5: 0.6780 - val_recall0.5: 0.3972 - val_tp0.7: 1240483.0000 - val_fp0.7: 547415.0000 - val_tn0.7: 21397672.0000 - val_fn0.7: 3028832.0000 - val_precision0.7: 0.6938 - val_recall0.7: 0.2906 - val_tp
    0.9: 699052.0000 - val_fp0.9: 209226.0000 - val_tn0.9: 21735862.0000 - val_fn0.9: 3570263.0000 - val_precision0.9: 0.7696 - val_recall0.9: 0.1637 - val_accuracy: 0.8711 - val_auc: 0.8155
    Epoch 48/50
    1280/1280 [==============================] - 3304s 3s/step - loss: 0.2335 - tp0.1: 7940859.5363 - fp0.1: 3183249.2857 - tn0.1: 41099674.0820 - fn0.1: 286865.7822 - precision0.1: 0.7138 - recall0.1: 0.9650 - tp0.3: 7641716.9641 - fp0.3:
     1582103.0671 - tn0.3: 42700833.3349 - fn0.3: 586008.3544 - precision0.3: 0.8288 - recall0.3: 0.9284 - tp0.5: 7316574.1686 - fp0.5: 906813.2022 - tn0.5: 43376114.1671 - fn0.5: 911151.1499 - precision0.5: 0.8900 - recall0.5: 0.8886 - tp
    0.7: 6893099.7447 - fp0.7: 495988.3950 - tn0.7: 43786957.1632 - fn0.7: 1334625.5738 - precision0.7: 0.9332 - recall0.7: 0.8372 - tp0.9: 5936967.9602 - fp0.9: 132958.7853 - tn0.9: 44149966.0359 - fn0.9: 2290757.3583 - precision0.9: 0.97
    82 - recall0.9: 0.7212 - accuracy: 0.9651 - auc: 0.9775 - val_loss: 0.8885 - val_tp0.1: 3090482.0000 - val_fp0.1: 1927063.0000 - val_tn0.1: 20147076.0000 - val_fn0.1: 1049776.0000 - val_precision0.1: 0.6159 - val_recall0.1: 0.7464 - va
    l_tp0.3: 2402659.0000 - val_fp0.3: 1358720.0000 - val_tn0.3: 20715420.0000 - val_fn0.3: 1737599.0000 - val_precision0.3: 0.6388 - val_recall0.3: 0.5803 - val_tp0.5: 2026443.0000 - val_fp0.5: 1121590.0000 - val_tn0.5: 20952544.0000 - va
    l_fn0.5: 2113815.0000 - val_precision0.5: 0.6437 - val_recall0.5: 0.4894 - val_tp0.7: 1587666.0000 - val_fp0.7: 757043.0000 - val_tn0.7: 21317100.0000 - val_fn0.7: 2552592.0000 - val_precision0.7: 0.6771 - val_recall0.7: 0.3835 - val_t
    p0.9: 946633.0000 - val_fp0.9: 302967.0000 - val_tn0.9: 21771180.0000 - val_fn0.9: 3193625.0000 - val_precision0.9: 0.7575 - val_recall0.9: 0.2286 - val_accuracy: 0.8766 - val_auc: 0.8420
    Epoch 49/50
    1280/1280 [==============================] - 3305s 3s/step - loss: 0.2225 - tp0.1: 7979679.4473 - fp0.1: 3114299.9914 - tn0.1: 41145497.0710 - fn0.1: 271189.6027 - precision0.1: 0.7186 - recall0.1: 0.9674 - tp0.3: 7686659.5777 - fp0.3:
     1542401.8501 - tn0.3: 42717409.8704 - fn0.3: 564209.4723 - precision0.3: 0.8334 - recall0.3: 0.9319 - tp0.5: 7371634.7557 - fp0.5: 895979.9727 - tn0.5: 43363790.1265 - fn0.5: 879234.2943 - precision0.5: 0.8923 - recall0.5: 0.8937 - tp
    0.7: 6947267.2155 - fp0.7: 481955.5433 - tn0.7: 43777844.2342 - fn0.7: 1303601.8345 - precision0.7: 0.9356 - recall0.7: 0.8420 - tp0.9: 6003926.6534 - fp0.9: 121081.5886 - tn0.9: 44138712.2896 - fn0.9: 2246942.3966 - precision0.9: 0.98
    05 - recall0.9: 0.7264 - accuracy: 0.9665 - auc: 0.9792 - val_loss: 0.9566 - val_tp0.1: 2861835.0000 - val_fp0.1: 1548477.0000 - val_tn0.1: 20587614.0000 - val_fn0.1: 1216470.0000 - val_precision0.1: 0.6489 - val_recall0.1: 0.7017 - va
    l_tp0.3: 2113828.0000 - val_fp0.3: 1067547.0000 - val_tn0.3: 21068548.0000 - val_fn0.3: 1964477.0000 - val_precision0.3: 0.6644 - val_recall0.3: 0.5183 - val_tp0.5: 1718632.0000 - val_fp0.5: 841787.0000 - val_tn0.5: 21294304.0000 - val
    _fn0.5: 2359673.0000 - val_precision0.5: 0.6712 - val_recall0.5: 0.4214 - val_tp0.7: 1302909.0000 - val_fp0.7: 555088.0000 - val_tn0.7: 21581000.0000 - val_fn0.7: 2775396.0000 - val_precision0.7: 0.7012 - val_recall0.7: 0.3195 - val_tp
    0.9: 773038.0000 - val_fp0.9: 218091.0000 - val_tn0.9: 21918012.0000 - val_fn0.9: 3305267.0000 - val_precision0.9: 0.7800 - val_recall0.9: 0.1895 - val_accuracy: 0.8779 - val_auc: 0.8265
    Epoch 50/50
    1280/1280 [==============================] - 3294s 3s/step - loss: 0.2301 - tp0.1: 8047942.4426 - fp0.1: 3082506.1288 - tn0.1: 41101922.8275 - fn0.1: 278310.9493 - precision0.1: 0.7219 - recall0.1: 0.9663 - tp0.3: 7760370.9899 - fp0.3:
     1521186.0101 - tn0.3: 42663221.2100 - fn0.3: 565882.4020 - precision0.3: 0.8352 - recall0.3: 0.9315 - tp0.5: 7453452.6776 - fp0.5: 892538.5199 - tn0.5: 43291840.9251 - fn0.5: 872800.7143 - precision0.5: 0.8922 - recall0.5: 0.8941 - tp
    0.7: 7032066.8470 - fp0.7: 486059.3411 - tn0.7: 43698327.0984 - fn0.7: 1294186.5449 - precision0.7: 0.9348 - recall0.7: 0.8430 - tp0.9: 6085441.6674 - fp0.9: 123868.2233 - tn0.9: 44060549.1827 - fn0.9: 2240811.7244 - precision0.9: 0.98
    05 - recall0.9: 0.7278 - accuracy: 0.9656 - auc: 0.9784 - val_loss: 0.7966 - val_tp0.1: 3023429.0000 - val_fp0.1: 1958644.0000 - val_tn0.1: 20321072.0000 - val_fn0.1: 911248.0000 - val_precision0.1: 0.6069 - val_recall0.1: 0.7684 - val
    _tp0.3: 2293385.0000 - val_fp0.3: 1327982.0000 - val_tn0.3: 20951742.0000 - val_fn0.3: 1641292.0000 - val_precision0.3: 0.6333 - val_recall0.3: 0.5829 - val_tp0.5: 1893004.0000 - val_fp0.5: 1112093.0000 - val_tn0.5: 21167626.0000 - val
    _fn0.5: 2041673.0000 - val_precision0.5: 0.6299 - val_recall0.5: 0.4811 - val_tp0.7: 1460348.0000 - val_fp0.7: 748911.0000 - val_tn0.7: 21530808.0000 - val_fn0.7: 2474329.0000 - val_precision0.7: 0.6610 - val_recall0.7: 0.3711 - val_tp
    0.9: 871307.0000 - val_fp0.9: 277884.0000 - val_tn0.9: 22001836.0000 - val_fn0.9: 3063370.0000 - val_precision0.9: 0.7582 - val_recall0.9: 0.2214 - val_accuracy: 0.8797 - val_auc: 0.8508
    600/600 [==============================] - 317s 527ms/step - loss: 0.9377 - tp0.1: 5542504.0000 - fp0.1: 4670897.0000 - tn0.1: 36924880.0000 - fn0.1: 2013729.0000 - precision0.1: 0.5427 - recall0.1: 0.7335 - tp0.3: 4216657.0000 - fp0.3
    : 3293157.0000 - tn0.3: 38302600.0000 - fn0.3: 3339576.0000 - precision0.3: 0.5615 - recall0.3: 0.5580 - tp0.5: 3498904.0000 - fp0.5: 2765631.0000 - tn0.5: 38830152.0000 - fn0.5: 4057329.0000 - precision0.5: 0.5585 - recall0.5: 0.4630
    - tp0.7: 2661409.0000 - fp0.7: 1815115.0000 - tn0.7: 39780628.0000 - fn0.7: 4894824.0000 - precision0.7: 0.5945 - recall0.7: 0.3522 - tp0.9: 1573540.0000 - fp0.9: 592773.0000 - tn0.9: 41002980.0000 - fn0.9: 5982693.0000 - precision0.9:
     0.7264 - recall0.9: 0.2082 - accuracy: 0.8612 - auc: 0.8217
    2021/03/18 20:42:08 INFO mlflow.projects: === Run (ID '3cec3f26ed2d4004978c4ec37c00fba0') succeeded ===
    (tf-nightly) [ye53nis@node117 drmed-git]$
    

2.3.6 Read out logs of Run 2

2.3.6.1 test dataset statistics
  • test data is not saved out automatically, but can be copied from the log above

    600/600 [==============================] - 317s 527ms/step - loss: 0.9377

    0.1 0.3 0.5 0.7 0.9
    tp 5542504.0000 4216657.0000 3498904.0000 2661409.0000 1573540.0000
    fp 4670897.0000 3293157.0000 2765631.0000 1815115.0000 592773.0000
    fn 2013729.0000 3339576.0000 4057329.0000 4894824.0000 5982693.0000
    tn 36924880.0000 38302600.0000 38830152.0000 39780628.0000 41002980.0000
    all 49152010 49151990 49152016 49151976 49151986
    precision 0.5427 0.5615 0.5585 0.5945 0.7264
    recall 0.7335 0.5580 0.4630 0.3522 0.2082

    accuracy: 0.8612 auc: 0.8217

    0.1 actual positive actual negative  
    pred positive 0.11276251 0.095029623 Prec: 0.5427
    pred negative 0.040969413 0.75123845  
      Recall: 0.7335    
          F1: 0.62383709
    0.3 actual positive actual negative  
    pred positive 0.085788124 0.066999464 Prec: 0.5615
    pred negative 0.067943861 0.77926855  
      Recall: 0.5580    
          F1: 0.55974453
    0.5 actual positive actual negative  
    pred positive 0.071185361 0.056266888 Prec: 0.5585
    pred negative 0.082546543 0.79000121  
      Recall: 0.4630    
          F1: 0.50628585
    0.7 actual positive actual negative  
    pred positive 0.054146531 0.036928627 Prec: 0.5945
    pred negative 0.099585498 0.80933934  
      Recall: 0.3522    
          F1: 0.44234266
    0.9 actual positive actual negative  
    pred positive 0.032013762 0.012060001 Prec: 0.7264
    pred negative 0.12171824 0.83420800  
      Recall: 0.2082    
          F1: 0.32363895
  • looks like pred_thresh=0.1 leads to the best results (Best F1 score)
2.3.6.2 prediction plots after each epoch

sadly, the random sample traces were not very useful, since the spikes were quite small…

2.3.6.3 Application 1 - git log, load modules, set parameters
  1. Git log, because some more code changes were made
    !git log -5
    
    commit 6586965ec900669bd641d69f85b4999050122502
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Tue Mar 23 20:26:11 2021 +0100
    
        Fix management of correlation failing 2
    
    commit ccdb3fb0887cce929987ef04408133917c02ce58
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Tue Mar 23 17:57:39 2021 +0100
    
        Merge plot_simulations and analyze_simulations 2
    
    commit 7951d725c9d53bbffd18c3fca8ccad05d528a368
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Sun Mar 21 12:49:47 2021 +0100
    
        Fix management of correlation failing; return nan
    
    commit 6c2146df58ba0af6a148745e7fafc7ff01a2e1b4
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Sun Mar 21 10:43:04 2021 +0100
    
        New management of correlation failing; return nan
    
    commit 940df93d2dacdb9c1951b6d8a8aa92f2c6043695
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Sat Mar 20 19:54:01 2021 +0100
    
        Merge plot_simulations and analyze_simulations
    
    %cd /beegfs/ye53nis/drmed-git
    
    /beegfs/ye53nis/drmed-git
    
    from pathlib import Path
    
    import sys
    import mlflow
    import matplotlib.pyplot as plt
    import numpy as np
    import pandas as pd
    import seaborn as sns
    mlflow.version.VERSION
    
    1.13.1
    
    sys.path.append('src/')
    from fluotracify.simulations import (
       import_simulation_from_csv as isfc,
       analyze_simulations as ans,
    )
    from fluotracify.training import build_model as bm, preprocess_data as ppd
    from fluotracify.applications import correlate, plots, correction
    from fluotracify.imports import ptu_utils as ptu
    
    import importlib
    importlib.reload(correction)
    
    <module 'fluotracify.applications.correction' from 'src/fluotracify/applications/correction.py'>
    
    folder = '/beegfs/ye53nis/saves/firstartifact_Nov2020_test'
    col_per_example = 3
    lab_thresh = 0.04
    pred_thresh = 0.1
    xunit = 1
    artifact = 0
    model_type = 1
    fwhm = 250
    run_id = '3cec3f26ed2d4004978c4ec37c00fba0'
    length_delimiter = 2**14
    
  2. now load the trained model
    mlflow.set_tracking_uri('file:///beegfs/ye53nis/drmed-git/data/mlruns')
    client = mlflow.tracking.MlflowClient(tracking_uri=mlflow.get_tracking_uri())
    model_path = client.download_artifacts(run_id=run_id,
                                           path='model')
    model_keras = mlflow.keras.load_model(model_uri=model_path,
                                          custom_objects={'binary_ce_dice':bm.binary_ce_dice_loss()})
    print(model_path, '\n', model_keras)
    
    /beegfs/ye53nis/drmed-git/3cec3f26ed2d4004978c4ec37c00fba0/artifacts/model
     <tensorflow.python.keras.engine.functional.Functional object at 0x2acdc9893190>
    
2.3.6.4 Application 2 - test data
  • test and train data were separated beforehand. Now, I only load test data.
    folder = '/beegfs/ye53nis/saves/firstartifact_Nov2020_test'
    dataset, _, nsamples, experiment_params = isfc.import_from_csv(
              folder=folder,
              header=12,
              frac_train=1,
              col_per_example=col_per_example,
              dropindex=None,
              dropcolumns=None)
    experiment_params
    
    train 0 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.069_set010.csv
    train 1 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D50_set002.csv
    train 2 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.4_set005.csv
    train 3 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.2_set005.csv
    train 4 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D3.0_set007.csv
    train 5 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D3.0_set002.csv
    train 6 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D50_set001.csv
    train 7 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.2_set007.csv
    train 8 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.6_set009.csv
    train 9 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D10_set002.csvtrain 10 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.08_set005.csv
    train 11 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.6_set008.csv
    train 12 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.1_set005.csv
    train 13 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.4_set008.csv
    train 14 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D10_set005.csv
    train 15 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D1.0_set006.csv
    train 16 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.069_set005.csv
    train 17 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D50_set003.csv
    train 18 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.1_set001.csv
    train 19 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.08_set003.csv
    train 20 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D1.0_set003.csv
    train 21 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D1.0_set005.csv
    train 22 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.2_set002.csv
    train 23 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.1_set002.csv
    train 24 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D3.0_set004.csv
    train 25 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.08_set001.csv
    train 26 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.069_set001.csv
    train 27 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D10_set001.csv
    train 28 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.6_set003.csv
    train 29 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/traces_brightclust_Nov2020_D0.4_set001.csv
    
    diffrates = experiment_params.loc['diffusion rate of molecules in micrometer^2 / s'].astype(np.float32)
    nmols = experiment_params.loc['number of fast molecules'].astype(np.float32)
    clusters = experiment_params.loc['diffusion rate of clusters in micrometer^2 / s'].astype(np.float32)
    
    dataset_sep = isfc.separate_data_and_labels(array=dataset,
                                                nsamples=nsamples,
                                                col_per_example=col_per_example)
    
    features = dataset_sep['0']
    labels_artifact = dataset_sep['1']
    labels_artifact_bool = labels_artifact > lab_thresh
    labels_puretrace = dataset_sep['2']
    
    The given DataFrame was split into 3 parts with shapes: [(16384, 3000), (16384, 3000), (16384, 3000)]
    
  • Let's correct the traces with the new model and correlate them!
    corr_out = ans.correlate_simulations_corrected_by_prediction(
        model=model_keras,
        lab_thresh=lab_thresh,
        pred_thresh=pred_thresh,
        artifact=artifact,
        model_type=model_type,
        experiment_params=experiment_params,
        nsamples=nsamples,
        features=features,
        labels_artifact=labels_artifact,
        labels_puretrace=labels_puretrace,
        save_as_csv=True)
    corr_out
    
    processed correlation of 3000 traces with correction by label
    processed correlation of 3000 traces with correction by prediction
    processed correlation of 3000 traces without correction
    processed correlation of pure 3000 traces
    
      Simulated \(D\) Simulated \(D\_{{clust}}\) nmol \(D\) in \(\frac{{\mu m\^2}}{{s}}\) \(\tau\_{{D}}\) in \(ms\) Trace lengths Traces used
    0 0.069 1.0 679.0 1.037836 10.860146 16384 corrupted without correction
    1 0.069 1.0 679.0 0.81403 13.846002 16384 corrupted without correction
    2 0.069 1.0 679.0 1.053716 10.696482 16384 corrupted without correction
    3 0.069 1.0 679.0 0.897022 12.564974 16384 corrupted without correction
    4 0.069 1.0 679.0 1.083577 10.401707 16384 corrupted without correction
    11995 0.4 0.1 1939.0 0.658104 17.126562 11187 corrected by prediction
    11996 0.4 0.1 1939.0 0.336804 33.464773 15720 corrected by prediction
    11997 0.4 0.1 1939.0 0.456482 24.691128 15292 corrected by prediction
    11998 0.4 0.1 1939.0 0.411872 27.365437 12516 corrected by prediction
    11999 0.4 0.1 1939.0 0.244884 46.026059 12986 corrected by prediction

    12000 rows × 7 columns

    corr_out = pd.read_csv(filepath_or_buffer='data/exp-210204-unet/2021-03-21_correlations.csv')
    corr_out[corr_out['$D$ in $\\frac{{\mu m^2}}{{s}}$'].isna()]
    
      Simulated \(D\) Simulated \(D\_{{clust}}\) nmol \(D\) in \(\frac{{\mu m\^2}}{{s}}\) \(\tau\_{{D}}\) in \(ms\) Trace lengths Traces used
    9109 50.0 0.01 2323.0 NaN NaN 1 corrected by prediction
    9118 50.0 0.01 2323.0 NaN NaN 6 corrected by prediction
    9131 50.0 0.01 2323.0 NaN NaN 26 corrected by prediction
    9134 50.0 0.01 2323.0 NaN NaN 3 corrected by prediction
    9141 50.0 0.01 2323.0 NaN NaN 6 corrected by prediction
    9144 50.0 0.01 2323.0 NaN NaN 17 corrected by prediction
    9163 50.0 0.01 2323.0 NaN NaN 24 corrected by prediction
    9170 50.0 0.01 2323.0 NaN NaN 27 corrected by prediction
    9171 50.0 0.01 2323.0 NaN NaN 30 corrected by prediction
    9174 50.0 0.01 2323.0 NaN NaN 8 corrected by prediction
    9183 50.0 0.01 2323.0 NaN NaN 13 corrected by prediction
    9187 50.0 0.01 2323.0 NaN NaN 29 corrected by prediction
    9199 50.0 0.01 2323.0 NaN NaN 18 corrected by prediction
    9909 10.0 0.01 1396.0 NaN NaN 24 corrected by prediction
    9922 10.0 0.01 1396.0 NaN NaN 7 corrected by prediction
    9931 10.0 0.01 1396.0 NaN NaN 16 corrected by prediction
    9978 10.0 0.01 1396.0 NaN NaN 31 corrected by prediction
    9998 10.0 0.01 1396.0 NaN NaN 9 corrected by prediction
  • We loaded the correlation results from the saved out .csv file and looked at where it failed. All the cases where basically all the trace was (wrongfully) removed, was at a very slow simulated cluster speed (0.01) and a rather high simulated speed of the molecules we are interested in (10 and 50).

    Awesome! Now let's try categorical plotting to examine the results in a more structured manner. First, plot the trace lengths.

    x = 'Trace lengths'
    
    g = sns.FacetGrid(data=corr_out,
                      col='Simulated $D$',
                      col_wrap=2,
                      sharex=True,
                      aspect=1.5,
                      height=5,
                      legend_out=True)
    g.map_dataframe(sns.boxplot,
          x=x,
          y='Traces used',
          hue='Simulated $D_{{clust}}$',
          palette='colorblind',
          showfliers=False)
    g.add_legend(title='Simulated $D_{{clust}}$')
    g.map_dataframe(sns.stripplot,
          x=x,
          y='Traces used',
          hue='Simulated $D_{{clust}}$',
          dodge=True,
          palette=sns.color_palette(['0.3']),
          size=4,
          jitter=0.2)
    g.set_xlabels(x)
    g.tight_layout()
    plt.show()
    
  • Then, plot the diffusion rates and transit times. Since they follow a log normal distribution, use a log scale.
    # '$D$ in $\\frac{{\mu m^2}}{{s}}$'
    # '$\\tau_{{D}}$ in $ms$'
    x = '$\\tau_{{D}}$ in $ms$'
    
    g = sns.FacetGrid(data=corr_out,
                      col='Simulated $D$',
                      col_wrap=2,
                      sharex=False,  # False for x=D or x=tau, True for x=Trace lengths
                      aspect=1.5,
                      height=5,
                      legend_out=True)
    g.map_dataframe(sns.boxplot,
          x=x,
          y='Traces used',
          hue='Simulated $D_{{clust}}$',
          palette='colorblind',
          showfliers=False).set(xscale = 'log')
    g.add_legend(title='Simulated $D_{{clust}}$')
    g.map_dataframe(sns.stripplot,
          x=x,
          y='Traces used',
          hue='Simulated $D_{{clust}}$',
          dodge=True,
          palette=sns.color_palette(['0.3']),
          size=4,
          jitter=0.2).set(xscale = 'log')
    g.set_xlabels(x)
    g.tight_layout()
    plt.show()
    

    2021-03-21_correlations_transit-times.png 2021-03-21_correlations_diffusion-rates.png 2021-03-21_correlations_trace-lengths.png

  • Now let's look at the corrections a little closer.
    corr_scatter = corr_out[corr_out['Traces used'].isin(['corrected by labels (control)', 'corrected by prediction'])]
    corr_scatter
    
      Simulated \(D\) Simulated \(D\_{{clust}}\) nmol \(D\) in \(\frac{{\mu m\^2}}{{s}}\) \(\tau\_{{D}}\) in \(ms\) Trace lengths Traces used
    6000 0.069 1.0 679.0 0.051680 218.095043 14468 corrected by labels (control)
    6001 0.069 1.0 679.0 0.050863 221.596899 15380 corrected by labels (control)
    6002 0.069 1.0 679.0 0.060130 187.443486 14908 corrected by labels (control)
    6003 0.069 1.0 679.0 0.087759 128.431237 15230 corrected by labels (control)
    6004 0.069 1.0 679.0 0.144906 77.781595 14930 corrected by labels (control)
    11995 0.400 0.1 1939.0 0.658104 17.126562 11187 corrected by prediction
    11996 0.400 0.1 1939.0 0.336804 33.464773 15720 corrected by prediction
    11997 0.400 0.1 1939.0 0.456482 24.691128 15292 corrected by prediction
    11998 0.400 0.1 1939.0 0.411872 27.365437 12516 corrected by prediction
    11999 0.400 0.1 1939.0 0.244884 46.026059 12986 corrected by prediction

    6000 rows × 7 columns

  • This scatterplot shows Diffusion rates / transit times against trace lengths.
    # '$D$ in $\\frac{{\mu m^2}}{{s}}$'
    # '$\\tau_{{D}}$ in $ms$'
    x = '$\\tau_{{D}}$ in $ms$'
    
    g = sns.FacetGrid(data=corr_scatter,
                      row='Simulated $D$',
                      col='Traces used',
                      sharex='row',
                      aspect=1,
                      height=4,
                      legend_out=True,
                      margin_titles=True)
    g.map_dataframe(sns.scatterplot,
          x=x,
          y='Trace lengths',
          hue='Simulated $D_{{clust}}$',
          palette='colorblind').set(xscale = 'log')
    g.add_legend(title='Simulated $D_{{clust}}$')
    g.set_xlabels(x)
    g.tight_layout()
    plt.show()
    
  • I concatenated the resulting 2 plots into 1 (left diffusion rates, right transit times). Here it can be nicely seen, that they are just two representations of the same dynamic phenomenon. 2021-03-21_correlations_tt-diffrate-tracelength-scatterplot.png
2.3.6.5 Application 3 - experimental data
  1. Workaround for the model prediction problem.
    test_features = np.zeros((2**14)).reshape(1, -1, 1)
    print(test_features.shape)
    predictions = model_keras.predict(test_features, verbose=0).flatten()
    predictions
    
    (1, 16384, 1)
    array([3.3430334e-15, 1.2115031e-08, 5.5057057e-09, ..., 3.8589034e-09,
           8.7170475e-09, 9.7742303e-14], dtype=float32)
    
    test_features = np.zeros((2**13)).reshape(1, -1, 1)
    print(test_features.shape)
    predictions = model_keras.predict(test_features, verbose=0).flatten()
    predictions
    
    (1, 8192, 1)
    array([3.3430334e-15, 1.2115031e-08, 5.5057057e-09, ..., 3.8588959e-09,
           8.7170475e-09, 9.7742493e-14], dtype=float32)
    
    path_tb_pex5_egfp = '/beegfs/ye53nis/data/Pablo_structured_experiment'
    pred_thresh = [0.1, 0.3, 0.5, 0.7, 0.9]
    length_delimiter = 2**13  # for U-Net
    bin_for_correlation = 1e6
    out = correction.correct_experimental_traces_from_ptu_by_unet_prediction(
      path_list=path_tb_pex5_egfp,
      model=model_keras,
      pred_thresh=pred_thresh,
      photon_count_bin=bin_for_correlation,
      ntraces=None,
      save_as_csv=True)
    out
    
    Loading dataset 1 from path /beegfs/ye53nis/data/Pablo_structured_experiment with bin=1e6. This can take a while...
    1 of 2: /beegfs/ye53nis/data/Pablo_structured_experiment/DiO LUV 10uM in 20 nM AF48822_T273s_1.ptu
    2 of 2: /beegfs/ye53nis/data/Pablo_structured_experiment/DiO LUV 10uM in 20 nM AF48821_T260s_1.ptu
    Processing correlation of unprocessed dataset 1
    Processing correlation with correction by prediction of dataset 1
    
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-186-8ad0b763f87d> in <module>
          3 length_delimiter = 2**13  # for U-Net
          4 bin_for_correlation = 1e6
    ----> 5 out = correction.correct_experimental_traces_from_ptu_by_unet_prediction(
          6   path_list=path_tb_pex5_egfp,
          7   model=model_keras,
    
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/correction.py in correct_experimental_traces_from_ptu_by_unet_prediction(path_list, model, pred_thresh, photon_count_bin, ntraces, save_as_csv)
        413             for thr in pred_thresh:
        414                 data['{}-pred-{}'.format(
    --> 415                     i, thr)] = correct_correlation_by_unet_prediction(
        416                         ntraces=ntraces,
        417                         traces_of_interest=ptu_1ms.astype(np.float64),
    
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/correction.py in correct_correlation_by_unet_prediction(ntraces, traces_of_interest, model, pred_thresh, fwhm, length_delimiter, traces_for_correlation, bin_for_correlation, verbose)
        215             ntraces_index=ntraces_index)
        216
    --> 217         predictions = model.predict(features_prepro, verbose=0)
        218         predictions = predictions.flatten()
        219
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py in predict(self, x, batch_size, verbose, steps, callbacks, max_queue_size, workers, use_multiprocessing)
       1661           for step in data_handler.steps():
       1662             callbacks.on_predict_batch_begin(step)
    -> 1663             tmp_batch_outputs = self.predict_function(iterator)
       1664             if data_handler.should_sync:
       1665               context.async_wait()
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
        816     tracing_count = self.experimental_get_tracing_count()
        817     with trace.Trace(self._name) as tm:
    --> 818       result = self._call(*args, **kwds)
        819       compiler = "xla" if self._jit_compile else "nonXla"
        820       new_tracing_count = self.experimental_get_tracing_count()
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
        851       # In this case we have not created variables on the first call. So we can
        852       # run the first trace but we should fail if variables are created.
    --> 853       results = self._stateful_fn(*args, **kwds)
        854       if self._created_variables:
        855         raise ValueError("Creating variables on a non-first call to a function"
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/function.py in __call__(self, *args, **kwargs)
       2990     with self._lock:
       2991       (graph_function,
    -> 2992        filtered_flat_args) = self._maybe_define_function(args, kwargs)
       2993     return graph_function._call_flat(
       2994         filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
       3408               self.input_signature is None and
       3409               call_context_key in self._function_cache.missed):
    -> 3410             return self._define_function_with_shape_relaxation(
       3411                 args, kwargs, flat_args, filtered_flat_args, cache_key_context)
       3412
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _define_function_with_shape_relaxation(self, args, kwargs, flat_args, filtered_flat_args, cache_key_context)
       3330           expand_composites=True)
       3331
    -> 3332     graph_function = self._create_graph_function(
       3333         args, kwargs, override_flat_arg_shapes=relaxed_arg_shapes)
       3334     self._function_cache.arg_relaxed[rank_only_cache_key] = graph_function
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
       3247     arg_names = base_arg_names + missing_arg_names
       3248     graph_function = ConcreteFunction(
    -> 3249         func_graph_module.func_graph_from_py_func(
       3250             self._name,
       3251             self._python_function,
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
        996         _, original_func = tf_decorator.unwrap(python_func)
        997
    --> 998       func_outputs = python_func(*func_args, **func_kwargs)
        999
       1000       # invariant: `func_outputs` contains only Tensors, CompositeTensors,
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
        610             xla_context.Exit()
        611         else:
    --> 612           out = weak_wrapped_fn().__wrapped__(*args, **kwds)
        613         return out
        614
    
    ~/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
        983           except Exception as e:  # pylint:disable=broad-except
        984             if hasattr(e, "ag_error_metadata"):
    --> 985               raise e.ag_error_metadata.to_exception(e)
        986             else:
        987               raise
    
    ValueError: in user code:
    
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1512 predict_function  *
            return step_function(self, iterator)
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1502 step_function  **
            outputs = model.distribute_strategy.run(run_step, args=(data,))
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:1262 run
            return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:2734 call_for_each_replica
            return self._call_for_each_replica(fn, args, kwargs)
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py:3423 _call_for_each_replica
            return fn(*args, **kwargs)
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1495 run_step  **
            outputs = model.predict_step(data)
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:1468 predict_step
            return self(x, training=False)
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py:1018 __call__
            input_spec.assert_input_compatibility(self.input_spec, inputs, self.name)
        /home/ye53nis/.conda/envs/tf-nightly/lib/python3.8/site-packages/tensorflow/python/keras/engine/input_spec.py:271 assert_input_compatibility
            raise ValueError('Input ' + str(input_index) +
    
        ValueError: Input 0 is incompatible with layer model: expected shape=(None, 16384, 1), found shape=(None, 8192, 1)
    
  2. Run correction for all experimental traces
    path_pex5_exp = ['/beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu', '/beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu']
    pred_thresh = [0.1, 0.3, 0.5, 0.7, 0.9]
    length_delimiter = 2**13  # for U-Net
    bin_for_correlation = 1e5
    out = correction.correct_experimental_traces_from_ptu_by_unet_prediction(
      path_list=path_pex5_exp,
      model=model_keras,
      pred_thresh=pred_thresh,
      photon_count_bin=bin_for_correlation,
      ntraces=400,
      save_as_csv=True)
    out
    
    Loading dataset 1 from path /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu with bin=1e6. This can take a while...
    1 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48891_T1082s_1.ptu
    2 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488129_T1537s_1.ptu
    3 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488210_T2505s_1.ptu
    4 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488199_T2375s_1.ptu
    5 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488171_T2040s_1.ptu
    6 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488229_T2733s_1.ptu
    7 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48890_T1070s_1.ptu
    8 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488113_T1352s_1.ptu
    9 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48883_T986s_1.ptu10 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488227_T2709s_1.ptu
    11 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488174_T2088s_1.ptu
    12 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488102_T1214s_1.ptu
    13 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488131_T1561s_1.ptu
    14 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48826_T302s_1.ptu
    15 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488162_T1932s_1.ptu
    16 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488190_T2267s_1.ptu
    17 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488189_T2255s_1.ptu
    18 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488208_T2481s_1.ptu
    19 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48827_T314s_1.ptu
    20 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488221_T2637s_1.ptu
    21 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488153_T1834s_1.ptu
    22 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488157_T1883s_1.ptu
    23 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488197_T2351s_1.ptu
    24 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48822_T254s_1.ptu
    25 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48816_T182s_1.ptu
    26 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488179_T2148s_1.ptu
    27 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488104_T1238s_1.ptu
    28 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48870_T833s_1.ptu
    29 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48857_T676s_1.ptu
    30 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488125_T1496s_1.ptu
    31 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48839_T459s_1.ptu
    32 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488135_T1610s_1.ptu
    33 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48850_T592s_1.ptu
    34 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488159_T1907s_1.ptu
    35 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48851_T604s_1.ptu
    36 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488158_T1895s_1.ptu
    37 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48878_T926s_1.ptu
    38 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48838_T447s_1.ptu
    39 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48869_T821s_1.ptu
    40 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48834_T399s_1.ptu
    41 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48843_T506s_1.ptu
    42 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48890_T1076s_1.ptu
    43 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488184_T2209s_1.ptu
    44 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48863_T748s_1.ptu
    45 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488103_T1232s_1.ptu
    46 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488181_T2159s_1.ptu
    47 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48859_T698s_1.ptu
    48 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48872_T855s_1.ptu
    49 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488150_T1789s_1.ptu
    50 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488241_T2877s_1.ptu
    51 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488204_T2434s_1.ptu
    52 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48877_T918s_1.ptu
    53 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488180_T2147s_1.ptu
    54 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488183_T2197s_1.ptu
    55 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488160_T1908s_1.ptu
    56 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488238_T2842s_1.ptu
    57 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488165_T1968s_1.ptu
    58 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488234_T2793s_1.ptu
    59 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48882_T979s_1.ptu
    60 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48896_T1148s_1.ptu
    61 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488107_T1280s_1.ptu
    62 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488138_T1654s_1.ptu
    63 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488124_T1484s_1.ptu
    64 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488187_T2245s_1.ptu
    65 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48874_T879s_1.ptu
    66 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488159_T1896s_1.ptu
    67 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488177_T2124s_1.ptu
    68 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488111_T1328s_1.ptu
    69 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48873_T867s_1.ptu
    70 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48881_T962s_1.ptu
    71 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48853_T628s_1.ptu
    72 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48842_T496s_1.ptu
    73 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48850_T590s_1.ptu
    74 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48899_T1184s_1.ptu
    75 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488196_T2339s_1.ptu
    76 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488164_T1956s_1.ptu
    77 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488217_T2589s_1.ptu
    78 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488218_T2601s_1.ptu
    79 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488119_T1418s_1.ptu
    80 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488206_T2458s_1.ptu
    81 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48866_T785s_1.ptu
    82 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488103_T1226s_1.ptu
    83 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48824_T278s_1.ptu
    84 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4887_T74s_1.ptu
    85 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488176_T2099s_1.ptu
    86 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48817_T194s_1.ptu
    87 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488113_T1346s_1.ptu
    88 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48858_T688s_1.ptu
    89 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488211_T2517s_1.ptu
    90 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48860_T712s_1.ptu
    91 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48833_T387s_1.ptu
    92 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48858_T686s_1.ptu
    93 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48813_T146s_1.ptu
    94 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488126_T1502s_1.ptu
    95 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48894_T1118s_1.ptu
    96 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488110_T1316s_1.ptu
    97 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488188_T2243s_1.ptu
    98 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488120_T1430s_1.ptu
    99 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488219_T2613s_1.ptu
    100 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488138_T1646s_1.ptu
    101 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488187_T2231s_1.ptu
    102 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488146_T1742s_1.ptu
    103 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488155_T1859s_1.ptu
    104 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488174_T2075s_1.ptu
    105 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488158_T1884s_1.ptu
    106 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488244_T2914s_1.ptu
    107 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4886_T62s_1.ptu
    108 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48885_T1015s_1.ptu
    109 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488165_T1979s_1.ptu
    110 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48857_T674s_1.ptu
    111 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488191_T2279s_1.ptu
    112 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488183_T2183s_1.ptu
    113 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488141_T1689s_1.ptu
    114 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488189_T2269s_1.ptu
    115 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48823_T266s_1.ptu
    116 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488209_T2493s_1.ptu
    117 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488136_T1630s_1.ptu
    118 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488185_T2221s_1.ptu
    119 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488179_T2135s_1.ptu
    120 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48866_T782s_1.ptu
    121 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488123_T1472s_1.ptu
    122 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488160_T1919s_1.ptu
    123 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48856_T664s_1.ptu
    124 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48830_T351s_1.ptu
    125 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48832_T375s_1.ptu
    126 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48859_T700s_1.ptu
    127 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488129_T1545s_1.ptu
    128 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488143_T1714s_1.ptu
    129 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48886_T1022s_1.ptu
    130 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488147_T1754s_1.ptu
    131 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488193_T2303s_1.ptu
    132 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488105_T1250s_1.ptu
    133 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488228_T2721s_1.ptu
    134 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48891_T1088s_1.ptu
    135 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488170_T2040s_1.ptu
    136 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48835_T411s_1.ptu
    137 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48888_T1046s_1.ptu
    138 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488123_T1466s_1.ptu
    139 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48876_T906s_1.ptu
    140 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48822_T255s_1.ptu
    141 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488145_T1730s_1.ptu
    142 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488114_T1358s_1.ptu
    143 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48824_T279s_1.ptu
    144 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48854_T638s_1.ptu
    145 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488247_T2949s_1.ptu
    146 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48844_T519s_1.ptu
    147 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48841_T482s_1.ptu
    148 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48828_T327s_1.ptu
    149 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48840_T471s_1.ptu
    150 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488225_T2685s_1.ptu
    151 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48831_T362s_1.ptu
    152 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48849_T578s_1.ptu
    153 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48861_T722s_1.ptu
    154 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48852_T614s_1.ptu
    155 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48852_T616s_1.ptu
    156 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488149_T1778s_1.ptu
    157 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488169_T2028s_1.ptu
    158 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4883_T26s_1.ptu
    159 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48881_T966s_1.ptu
    160 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48837_T435s_1.ptu
    161 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488139_T1666s_1.ptu
    162 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488194_T2315s_1.ptu
    163 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488205_T2446s_1.ptu
    164 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48818_T206s_1.ptu
    165 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488128_T1533s_1.ptu
    166 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48851_T602s_1.ptu
    167 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488116_T1388s_1.ptu
    168 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488184_T2196s_1.ptu
    169 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488117_T1394s_1.ptu
    170 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48869_T818s_1.ptu
    171 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48895_T1130s_1.ptu
    172 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488173_T2076s_1.ptu
    173 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488106_T1262s_1.ptu
    174 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488249_T2993s_1.ptu
    175 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48889_T1063s_1.ptu
    176 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48845_T531s_1.ptu
    177 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488117_T1400s_1.ptu
    178 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488192_T2291s_1.ptu
    179 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488166_T1992s_1.ptu
    180 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48896_T1142s_1.ptu
    181 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48894_T1124s_1.ptu
    182 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488177_T2111s_1.ptu
    183 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48862_T736s_1.ptu
    184 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488118_T1412s_1.ptu
    185 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48892_T1100s_1.ptu
    186 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48847_T555s_1.ptu
    187 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488223_T2661s_1.ptu
    188 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48853_T626s_1.ptu
    189 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488136_T1622s_1.ptu
    190 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488151_T1810s_1.ptu
    191 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488245_T2925s_1.ptu
    192 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48879_T942s_1.ptu
    193 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488121_T1448s_1.ptu
    194 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48868_T809s_1.ptu
    195 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488201_T2398s_1.ptu
    196 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488140_T1670s_1.ptu
    197 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488250_T3006s_1.ptu
    198 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4884_T38s_1.ptu
    199 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488127_T1514s_1.ptu
    200 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488104_T1244s_1.ptu
    201 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488181_T2172s_1.ptu
    202 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488115_T1370s_1.ptu
    203 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488140_T1678s_1.ptu
    204 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48855_T652s_1.ptu
    205 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48884_T1003s_1.ptu
    206 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48846_T544s_1.ptu
    207 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488131_T1569s_1.ptu
    208 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488236_T2817s_1.ptu
    209 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488178_T2123s_1.ptu
    210 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48873_T869s_1.ptu
    211 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488164_T1968s_1.ptu
    212 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488249_T2973s_1.ptu
    213 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488151_T1801s_1.ptu
    214 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488235_T2805s_1.ptu
    215 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488170_T2028s_1.ptu
    216 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488180_T2160s_1.ptu
    217 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488135_T1618s_1.ptu
    218 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488190_T2281s_1.ptu
    219 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48864_T758s_1.ptu
    220 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48867_T794s_1.ptu
    221 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48875_T891s_1.ptu
    222 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48871_T842s_1.ptu
    223 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488213_T2541s_1.ptu
    224 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488182_T2184s_1.ptu
    225 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48841_T484s_1.ptu
    226 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488171_T2052s_1.ptu
    227 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488153_T1825s_1.ptu
    228 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488132_T1581s_1.ptu
    229 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488147_T1762s_1.ptu
    230 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488110_T1310s_1.ptu
    231 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48885_T1010s_1.ptu
    232 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48825_T290s_1.ptu
    233 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488168_T2004s_1.ptu
    234 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48820_T230s_1.ptu
    235 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488162_T1944s_1.ptu
    236 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488203_T2422s_1.ptu
    237 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488152_T1822s_1.ptu
    238 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48821_T243s_1.ptu
    239 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488167_T2004s_1.ptu
    240 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48837_T436s_1.ptu
    241 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488166_T1980s_1.ptu
    242 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488200_T2386s_1.ptu
    243 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488122_T1454s_1.ptu
    244 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48843_T508s_1.ptu
    245 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488102_T1220s_1.ptu
    246 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48865_T773s_1.ptu
    247 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488161_T1920s_1.ptu
    248 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48823_T267s_1.ptu
    249 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48863_T746s_1.ptu
    250 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488132_T1573s_1.ptu
    251 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48867_T797s_1.ptu
    252 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488144_T1726s_1.ptu
    253 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488145_T1738s_1.ptu
    254 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488169_T2016s_1.ptu
    255 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48889_T1058s_1.ptu
    256 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48860_T710s_1.ptu
    257 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48831_T363s_1.ptu
    258 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488215_T2565s_1.ptu
    259 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48878_T930s_1.ptu
    260 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48865_T770s_1.ptu
    261 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48870_T830s_1.ptu
    262 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488114_T1364s_1.ptu
    263 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488148_T1774s_1.ptu
    264 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488224_T2673s_1.ptu
    265 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488172_T2052s_1.ptu
    266 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488167_T1992s_1.ptu
    267 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488100_T1190s_1.ptu
    268 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48821_T242s_1.ptu
    269 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488240_T2865s_1.ptu
    270 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488142_T1701s_1.ptu
    271 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488130_T1549s_1.ptu
    272 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48844_T520s_1.ptu
    273 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48861_T724s_1.ptu
    274 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48814_T158s_1.ptu
    275 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488163_T1956s_1.ptu
    276 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488173_T2064s_1.ptu
    277 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4885_T50s_1.ptu
    278 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48849_T580s_1.ptu
    279 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48840_T472s_1.ptu
    280 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48883_T991s_1.ptu
    281 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488247_T2969s_1.ptu
    282 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488150_T1798s_1.ptu
    283 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48848_T566s_1.ptu
    284 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48897_T1160s_1.ptu
    285 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488118_T1406s_1.ptu
    286 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4889_T98s_1.ptu
    287 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48812_T134s_1.ptu
    288 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488222_T2649s_1.ptu
    289 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48887_T1039s_1.ptu
    290 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48879_T938s_1.ptu
    291 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48888_T1051s_1.ptu
    292 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48876_T902s_1.ptu
    293 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48819_T218s_1.ptu
    294 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48893_T1112s_1.ptu
    295 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488121_T1442s_1.ptu
    296 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488101_T1208s_1.ptu
    297 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488109_T1304s_1.ptu
    298 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488128_T1526s_1.ptu
    299 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488207_T2470s_1.ptu
    300 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488246_T2937s_1.ptu
    301 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488242_T2890s_1.ptu
    302 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48854_T640s_1.ptu
    303 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48847_T554s_1.ptu
    304 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488108_T1292s_1.ptu
    305 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488134_T1597s_1.ptu
    306 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4888_T86s_1.ptu
    307 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48872_T857s_1.ptu
    308 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488149_T1786s_1.ptu
    309 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48825_T291s_1.ptu
    310 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488182_T2171s_1.ptu
    311 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488156_T1871s_1.ptu
    312 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488220_T2624s_1.ptu
    313 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488248_T2981s_1.ptu
    314 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48897_T1154s_1.ptu
    315 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48848_T568s_1.ptu
    316 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488175_T2100s_1.ptu
    317 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488154_T1847s_1.ptu
    318 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488245_T2945s_1.ptu
    319 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488139_T1658s_1.ptu
    320 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48830_T350s_1.ptu
    321 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488175_T2087s_1.ptu
    322 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48856_T662s_1.ptu
    323 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488230_T2745s_1.ptu
    324 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48875_T894s_1.ptu
    325 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488127_T1521s_1.ptu
    326 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488195_T2327s_1.ptu
    327 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48884_T998s_1.ptu
    328 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488186_T2219s_1.ptu
    329 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48882_T974s_1.ptu
    330 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488134_T1606s_1.ptu
    331 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488108_T1286s_1.ptu
    332 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48827_T315s_1.ptu
    333 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488214_T2553s_1.ptu
    334 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488163_T1944s_1.ptu
    335 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48839_T460s_1.ptu
    336 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488186_T2233s_1.ptu
    337 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488243_T2902s_1.ptu
    338 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488137_T1642s_1.ptu
    339 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488156_T1861s_1.ptu
    340 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4882_T14s_1.ptu
    341 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488142_T1694s_1.ptu
    342 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48811_T123s_1.ptu
    343 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488248_T2961s_1.ptu
    344 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48895_T1136s_1.ptu
    345 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48826_T303s_1.ptu
    346 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488191_T2293s_1.ptu
    347 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488137_T1634s_1.ptu
    348 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488133_T1585s_1.ptu
    349 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488216_T2577s_1.ptu
    350 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488107_T1274s_1.ptu
    351 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488130_T1557s_1.ptu
    352 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488120_T1436s_1.ptu
    353 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48868_T806s_1.ptu
    354 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488125_T1490s_1.ptu
    355 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48832_T374s_1.ptu
    356 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488119_T1424s_1.ptu
    357 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48877_T914s_1.ptu
    358 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488188_T2257s_1.ptu
    359 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48836_T423s_1.ptu
    360 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488176_T2112s_1.ptu
    361 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488112_T1334s_1.ptu
    362 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48815_T170s_1.ptu
    363 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48838_T448s_1.ptu
    364 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488178_T2136s_1.ptu
    365 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488198_T2363s_1.ptu
    366 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48855_T650s_1.ptu
    367 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48899_T1178s_1.ptu
    368 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488161_T1931s_1.ptu
    369 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488246_T2957s_1.ptu
    370 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488231_T2757s_1.ptu
    371 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488122_T1460s_1.ptu
    372 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488157_T1873s_1.ptu
    373 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488194_T2329s_1.ptu
    374 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48846_T542s_1.ptu
    375 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488144_T1718s_1.ptu
    376 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488111_T1322s_1.ptu
    377 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488143_T1705s_1.ptu
    378 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48874_T882s_1.ptu
    379 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488124_T1478s_1.ptu
    380 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488106_T1268s_1.ptu
    381 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48892_T1094s_1.ptu
    382 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488115_T1376s_1.ptu
    383 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48864_T761s_1.ptu
    384 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488239_T2853s_1.ptu
    385 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48893_T1106s_1.ptu
    386 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488154_T1837s_1.ptu
    387 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488105_T1256s_1.ptu
    388 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48829_T338s_1.ptu
    389 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48871_T845s_1.ptu
    390 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48898_T1172s_1.ptu
    391 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48880_T954s_1.ptu
    392 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48811_T122s_1.ptu
    393 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488141_T1682s_1.ptu
    394 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488237_T2829s_1.ptu
    395 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48810_T110s_1.ptu
    396 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48898_T1166s_1.ptu
    397 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48812_T135s_1.ptu
    398 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48887_T1034s_1.ptu
    399 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488202_T2410s_1.ptu
    400 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48828_T326s_1.ptu
    401 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48886_T1027s_1.ptu
    Different binning was chosen for correlation. Loading dataset 1 with bin=100000.0. This can take a while...
    Processing correlation of unprocessed dataset 1
    Processing correlation with correction by prediction of dataset 1
    
  3. Not all of the log was saved (e.g. loading folder alldirtyptu). Load CSV
    corr_out = pd.read_csv(filepath_or_buffer='data/exp-210204-unet/2021-03-24_correlations.csv')
    corr_out
    
      \(D\) in \(\frac{{\mu m\^2}}{{s}}\) \(\tau\_{{D}}\) in \(ms\) Trace lengths folder\id-traces\used Photon count bin for correlation in \(ns\)
    0 19.078159 0.590783 8192.0 0-orig 100000.0
    1 20.314897 0.554817 8192.0 0-orig 100000.0
    2 21.007772 0.536518 8192.0 0-orig 100000.0
    3 22.716317 0.496166 8192.0 0-orig 100000.0
    4 24.089236 0.467888 8192.0 0-orig 100000.0
    4795 0.062475 180.409905 8017.0 1-pred-0.9 100000.0
    4796 0.054750 205.864659 7976.0 1-pred-0.9 100000.0
    4797 0.265582 42.439007 8060.0 1-pred-0.9 100000.0
    4798 0.148929 75.680566 8077.0 1-pred-0.9 100000.0
    4799 0.331785 33.970940 8091.0 1-pred-0.9 100000.0

    4800 rows × 5 columns

    looks good! Open question: why were metadata not saved? Since I started handling multipletau failures better (they now return np.nan values for the correlations instead of just skipping the correlation) the shapes of metadata and data should match…

  4. Check out nan values
    corr_out[corr_out['$D$ in $\\frac{{\mu m^2}}{{s}}$'].isna()]
    
      \(D\) in \(\frac{{\mu m\^2}}{{s}}\) \(\tau\_{{D}}\) in \(ms\) Trace lengths folder\id-traces\used Photon count bin for correlation in \(ns\)
    402 NaN NaN 3.0 0-pred-0.1 100000.0
    410 NaN NaN 3.0 0-pred-0.1 100000.0
    428 NaN NaN 0.0 0-pred-0.1 100000.0
    481 NaN NaN 3.0 0-pred-0.1 100000.0
    527 NaN NaN 3.0 0-pred-0.1 100000.0
    530 NaN NaN 1.0 0-pred-0.1 100000.0
    534 NaN NaN 0.0 0-pred-0.1 100000.0
    552 NaN NaN 1.0 0-pred-0.1 100000.0
    553 NaN NaN 1.0 0-pred-0.1 100000.0
    557 NaN NaN 2.0 0-pred-0.1 100000.0
    558 NaN NaN 2.0 0-pred-0.1 100000.0
    563 NaN NaN 2.0 0-pred-0.1 100000.0
    568 NaN NaN 1.0 0-pred-0.1 100000.0
    572 NaN NaN 0.0 0-pred-0.1 100000.0
    578 NaN NaN 1.0 0-pred-0.1 100000.0
    589 NaN NaN 2.0 0-pred-0.1 100000.0
    623 NaN NaN 3.0 0-pred-0.1 100000.0
    634 NaN NaN 1.0 0-pred-0.1 100000.0
    638 NaN NaN 1.0 0-pred-0.1 100000.0
    651 NaN NaN 2.0 0-pred-0.1 100000.0
    655 NaN NaN 3.0 0-pred-0.1 100000.0
    656 NaN NaN 0.0 0-pred-0.1 100000.0
    660 NaN NaN 0.0 0-pred-0.1 100000.0
    691 NaN NaN 0.0 0-pred-0.1 100000.0
    694 NaN NaN 0.0 0-pred-0.1 100000.0
    699 NaN NaN 2.0 0-pred-0.1 100000.0
    715 NaN NaN 2.0 0-pred-0.1 100000.0
    721 NaN NaN 0.0 0-pred-0.1 100000.0
    725 NaN NaN 1.0 0-pred-0.1 100000.0
    764 NaN NaN 2.0 0-pred-0.1 100000.0
    799 NaN NaN 1.0 0-pred-0.1 100000.0

    We see, that all of the 31 NaNs happened in the "clean" dataset with a prediction threshold of 0.1 and all were presumably caused by the insufficient trace length, since multipletau needs in this case 32 time steps to correlate the trace.

  5. Refactor folder-idtraces-used into 2 columns
    corr_out['folder_id-traces_used']
    
    0           0-orig
    1           0-orig
    2           0-orig
    3           0-orig
    4           0-orig
               ...
    4795    1-pred-0.9
    4796    1-pred-0.9
    4797    1-pred-0.9
    4798    1-pred-0.9
    4799    1-pred-0.9
    Name: folder_id-traces_used, Length: 4800, dtype: object
    
    corr_out[['Folder ID', 'Traces used']] = corr_out['folder_id-traces_used'].str.split(pat='-', n=1, expand=True)
    
    corr_out
    
      \(D\) in \(\frac{{\mu m\^2}}{{s}}\) \(\tau\_{{D}}\) in \(ms\) Trace lengths folder\id-traces\used Photon count bin for correlation in \(ns\) Folder ID Traces used
    0 19.078159 0.590783 8192.0 0-orig 100000.0 0 orig
    1 20.314897 0.554817 8192.0 0-orig 100000.0 0 orig
    2 21.007772 0.536518 8192.0 0-orig 100000.0 0 orig
    3 22.716317 0.496166 8192.0 0-orig 100000.0 0 orig
    4 24.089236 0.467888 8192.0 0-orig 100000.0 0 orig
    4795 0.062475 180.409905 8017.0 1-pred-0.9 100000.0 1 pred-0.9
    4796 0.054750 205.864659 7976.0 1-pred-0.9 100000.0 1 pred-0.9
    4797 0.265582 42.439007 8060.0 1-pred-0.9 100000.0 1 pred-0.9
    4798 0.148929 75.680566 8077.0 1-pred-0.9 100000.0 1 pred-0.9
    4799 0.331785 33.970940 8091.0 1-pred-0.9 100000.0 1 pred-0.9

    4800 rows × 7 columns

  6. plots
    x = 'Trace lengths'
    
    g = sns.FacetGrid(data=corr_out,
                      col='Folder ID',
                      col_wrap=1,
                      sharex=True,
                      aspect=1.5,
                      height=5,
                      legend_out=True)
    g.map_dataframe(sns.boxplot,
          x=x,
          y='Traces used',
          palette='colorblind',
          showfliers=False)
    g.map_dataframe(sns.stripplot,
          x=x,
          y='Traces used',
          dodge=True,
          palette=sns.color_palette(['0.3']),
          size=4,
          jitter=0.2)
    g.set_xlabels(x)
    g.tight_layout()
    plt.show()
    

    Trace lengths: 2021-03-24_correlations_trace-lengths.png

    # '$D$ in $\\frac{{\mu m^2}}{{s}}$'
    # '$\\tau_{{D}}$ in $ms$'
    x = '$D$ in $\\frac{{\mu m^2}}{{s}}$'
    
    g = sns.FacetGrid(data=corr_out,
                      col='Folder ID',
                      col_wrap=1,
                      sharex=True,
                      aspect=1.5,
                      height=5,
                      legend_out=True)
    g.map_dataframe(sns.boxplot,
          x=x,
          y='Traces used',
          palette='colorblind',
          showfliers=False).set(xscale = 'log')
    g.map_dataframe(sns.stripplot,
          x=x,
          y='Traces used',
          dodge=True,
          palette=sns.color_palette(['0.3']),
          size=4,
          jitter=0.2).set(xscale = 'log')
    g.set_xlabels(x)
    g.tight_layout()
    plt.show()
    

    D: 2021-03-24_correlations_diffusion-rates.png tau: 2021-03-24_correlations_transit-times.png

    This scatterplot shows Diffusion rates / transit times against trace lengths. I used a subsample to avoid overplotting.

    # '$D$ in $\\frac{{\mu m^2}}{{s}}$'
    # '$\\tau_{{D}}$ in $ms$'
    x = '$\\tau_{{D}}$ in $ms$'
    
    g = sns.FacetGrid(data=corr_out,
                      row='Traces used',
                      col='Folder ID',
                      hue='Traces used',
                      sharex=True,
                      sharey=True,
                      aspect=1.5,
                      height=3.5,
                      margin_titles=True,
                      legend_out=True)
    g.map_dataframe(sns.scatterplot,
          x=x,
          y='Trace lengths',
          palette='colorblind').set(xscale = 'log')
    g.add_legend(title='Traces used')
    g.set_xlabels(x)
    g.tight_layout()
    plt.show()
    
    # '$D$ in $\\frac{{\mu m^2}}{{s}}$'
    # '$\\tau_{{D}}$ in $ms$'
    x = '$\\tau_{{D}}$ in $ms$'
    
    g = sns.FacetGrid(data=corr_out.sample(1000),
                      row='Folder ID',
                      hue='Traces used',
                      hue_order=['orig', 'pred-0.1', 'pred-0.3', 'pred-0.5', 'pred-0.7', 'pred-0.9'],
                      sharex=True,
                      sharey=True,
                      aspect=1.5,
                      height=4,
                      margin_titles=True,
                      legend_out=True)
    g.map_dataframe(sns.scatterplot,
          x=x,
          y='Trace lengths',
          palette='colorblind').set(xscale = 'log')
    g.add_legend(title='Traces used')
    g.set_xlabels(x)
    g.tight_layout()
    plt.show()
    

    Scatterplot on subsample: 2021-03-24_correlations_scatter-subsample.png Scatterplot on full dataset, but categorical: 2021-03-24_correlations_scatter-full.png

2.3.7 use MLFLOW to compare run 1 and run 2

  1. get the experiment ID
    conda activate tf-nightly
    cd Programme/drmed-git
    export MLFLOW_EXPERIMENT_NAME=exp-210204-unet
    
    export MLFLOW_TRACKING_URI=file:/data/mlruns
    mlflow experiments list
    
      Experiment Id  Name             Artifact Location
    ---------------  ---------------  -------------------
                  2  exp-210124-test  file:.
                  3  exp-210204-unet  file:.
    
    mlflow runs list --experiment-id 3
    
    Date                     Name    ID
    -----------------------  ------  --------------------------------
    2021-03-16 22:26:12 CET          3cec3f26ed2d4004978c4ec37c00fba0
    2021-02-05 18:23:50 CET          b9935d1e554c423fb2852242f4c4504c
    
  2. start tensorboard for the runs
    export EXP1=b9935d1e554c423fb2852242f4c4504c
    export EXP2=3cec3f26ed2d4004978c4ec37c00fba0
    tensorboard --logdir=data/mlruns/3/$EXP2/artifacts/tensorboard_logs
    
                     
    sh-5.1$ sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:          
    ye53nis@login01's password:              
    bind: Address already in use        
    Last login: Fri Apr 2 23:45:47 2021 from 10.231.191.246

auc and val-auc: auc-and-val-auc.png

loss and val-loss: loss-and-val-loss.png

epoch_precision_and_recall.svg

run 1 distribution and histogram of final layer run1_distribution_final-layer.png run1_histogram_final-layer.png

run 2 distribution and histogram of final layer run2_distribution_final-layer.png run2_histogram_final-layer.png

2.4 exp-210807-hparams

2.4.1 Connect

2.4.1.1 GPU node for script execution
  1. Setup tmux
    rm: cannot remove 'home/lex.tmux-local-socket-remote-machine': No such file or directory
    ye53nis@ara-login01.rz.uni-jena.de's password:              
    /tmp/tmux-67339/default                
    > ye53nis@ara-login01.rz.uni-jena.de's password:            
  2. first, connect with the GPU node in the high performance cluster
    cd /
    srun -p gpu_v100 --time=5-10:00:00 --ntasks-per-node=12 --mem-per-cpu=4000 --gres=gpu:1 --pty bash
    
    (base) [ye53nis@node130 /]$
    
  3. Load CUDA and cuDNN in the version compatible to your tensorflow library (see https://www.tensorflow.org/install/source#gpu)
    module load nvidia/cuda/11.2
    module load nvidia/cudnn/8.1
    module list
    
  4. Branch out git branch exp-210807-hparams from main (done via magit) and make sure you are on the correct branch
    cd /beegfs/ye53nis/drmed-git
    git checkout exp-210807-hparams
    
    (base) [ye53nis@node130 drmed-git]$ git checkout exp-210807-hparams
    Checking out files: 100% (148/148), done.
    M       src/nanosimpy
    Branch exp-210807-hparams set up to track remote branch exp-210807-hparams from origin.
    Switched to a new branch 'exp-210807-hparams'
    \nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/post-checkout.\n
    (base) [ye53nis@node130 drmed-git]$
    
  5. Side quest: remove last remnants of failed git LFS experiments
    cat .git/hooks/post-checkout
    rm .git/hooks/post-checkout
    rm .git/hooks/pre-push
    
    (base) [ye53nis@node130 drmed-git]$ cat .git/hooks/post-checkout
    #!/bin/sh
    command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/post-checkout.\n"; ex
    it 2; }
    git lfs post-checkout "$@"
    (base) [ye53nis@node130 drmed-git]$ rm .git/hooks/post-checkout
    (base) [ye53nis@node130 drmed-git]$ rm .git/hooks/pre-push
    (base) [ye53nis@node130 drmed-git]$ git push origin exp-210807-hparams
    Username for 'https://github.com': aseltmann
    Password for 'https://aseltmann@github.com':
    Everything up-to-date
    (base) [ye53nis@node130 drmed-git]$
    
  6. Load conda environment
    conda activate tf
    
    (tf) [ye53nis@node130 drmed-git]$
    
  7. define MLflow environment variables and create log directory
    cd /beegfs/ye53nis/drmed-git
    export MLFLOW_EXPERIMENT_NAME=exp-210807-hparams
    export MLFLOW_TRACKING_URI=file:/beegfs/ye53nis/drmed-git/data/mlruns
    mkdir data/exp-210807-hparams
    
    (tf) [ye53nis@node130 drmed-git]$
    
2.4.1.2 Node for running Jupyter
  1. customize the output folder using the following org-mode variable:
    (setq org-babel-jupyter-resource-directory "./data/exp-210807-hparams/jupyter")
    
    ./data/exp-210807-hparams/jupyter
    
  2. Set up tmux (if we haven't done that before) (#+CALL: setup-tmux[:session local])
         
    sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:
    > ye53nis@ara-login01.rz.uni-jena.de's password:
  3. Request compute node
    cd /
    srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
    
  4. Start Jupyter Lab (#+CALL: jpt-tmux[:session jpmux])
    (tf) [ye53nis@node144 /]$ jupyter lab --no-browser --port=$PORT
    [I 2021-12-14 13:02:25.053 ServerApp] jupyterlab | extension was successfully linked.
    [I 2021-12-14 13:02:30.949 ServerApp] nbclassic | extension was successfully linked.
    [I 2021-12-14 13:02:31.454 LabApp] JupyterLab extension loaded from /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/jupyterlab
    [I 2021-12-14 13:02:31.455 LabApp] JupyterLab application directory is /home/ye53nis/.conda/envs/tf/share/jupyter/lab
    [I 2021-12-14 13:02:31.464 ServerApp] jupyterlab | extension was successfully loaded.
    [I 2021-12-14 13:02:31.591 ServerApp] nbclassic | extension was successfully loaded.
    [I 2021-12-14 13:02:31.592 ServerApp] Serving notebooks from local directory: /
    [I 2021-12-14 13:02:31.592 ServerApp] Jupyter Server 1.4.1 is running at:
    [I 2021-12-14 13:02:31.592 ServerApp] http://localhost:9999/lab?token=d56ed5d84f4a58d80eccfc44527559c44840ce3fd540c1b2
    [I 2021-12-14 13:02:31.592 ServerApp]  or http://127.0.0.1:9999/lab?token=d56ed5d84f4a58d80eccfc44527559c44840ce3fd540c1b2
    [I 2021-12-14 13:02:31.592 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
    [C 2021-12-14 13:02:31.625 ServerApp]
    
        To access the server, open this file in a browser:
            file:///home/ye53nis/.local/share/jupyter/runtime/jpserver-92394-open.html
        Or copy and paste one of these URLs:
            http://localhost:9999/lab?token=d56ed5d84f4a58d80eccfc44527559c44840ce3fd540c1b2
         or http://127.0.0.1:9999/lab?token=d56ed5d84f4a58d80eccfc44527559c44840ce3fd540c1b2
    
  5. Create SSH Tunnel for jupyter lab to the local computer (e.g. #+CALL: ssh-tunnel(port="9999", node="node152"))
                     
    sh-5.1$ sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:          
    ye53nis@node144's password:              
    Last login: Tue Dec 14 22:39:42 2021 from login01.ara
  6. I started a Python3 kernel using jupyter-server-list-kernels. Then I added the kernel ID to the :PROPERTIES: drawer of this (and following) subtrees.
    python3           c4f3acce-60c4-489d-922c-407da110fd6a   a few seconds ago    idle       1
    
  7. Test (#+CALL: jp-metadata(_long='True)):
    No of CPUs in system: 72
    No of CPUs the current process can use: 24
    load average: (17.06, 17.03, 18.02)
    os.uname():  posix.uname_result(sysname='Linux', nodename='node162', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
    PID of process: 40969
    RAM total: 199G, RAM used: 13G, RAM free: 164G
    the current directory: /
    My disk usage:
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/sda1             50G  3.8G   47G   8% /
    devtmpfs              94G     0   94G   0% /dev
    tmpfs                 94G  8.6M   94G   1% /dev/shm
    tmpfs                 94G  115M   94G   1% /run
    tmpfs                 94G     0   94G   0% /sys/fs/cgroup
    nfs01-ib:/home        80T   67T   14T  83% /home
    nfs03-ib:/pool/work  100T   78T   22T  79% /nfsdata
    nfs02-ib:/data01      88T   71T   17T  82% /data01
    nfs01-ib:/cluster    2.0T  464G  1.6T  23% /cluster
    /dev/sda5            2.0G   34M  2.0G   2% /tmp
    /dev/sda3            6.0G  428M  5.6G   7% /var
    /dev/sda6            169G   33M  169G   1% /local
    beegfs_nodev         524T  484T   41T  93% /beegfs
    tmpfs                 19G     0   19G   0% /run/user/67339# packages in environment at /home/ye53nis/.conda/envs/tf:
    #
    # Name                    Version                   Build  Channel
    _libgcc_mutex             0.1                        main
    _openmp_mutex             4.5                       1_gnu
    absl-py                   0.13.0                   pypi_0    pypi
    alembic                   1.4.1                    pypi_0    pypi
    anyio                     2.2.0            py39h06a4308_1
    argon2-cffi               20.1.0           py39h27cfd23_1
    asteval                   0.9.25                   pypi_0    pypi
    astunparse                1.6.3                    pypi_0    pypi
    async_generator           1.10               pyhd3eb1b0_0
    attrs                     21.2.0             pyhd3eb1b0_0
    babel                     2.9.1              pyhd3eb1b0_0
    backcall                  0.2.0              pyhd3eb1b0_0
    bleach                    3.3.1              pyhd3eb1b0_0
    brotlipy                  0.7.0           py39h27cfd23_1003
    ca-certificates           2021.7.5             h06a4308_1
    cachetools                4.2.2                    pypi_0    pypi
    certifi                   2021.5.30        py39h06a4308_0
    cffi                      1.14.6           py39h400218f_0
    chardet                   4.0.0           py39h06a4308_1003
    click                     8.0.1                    pypi_0    pypi
    cloudpickle               1.6.0                    pypi_0    pypi
    cryptography              3.4.7            py39hd23ed53_0
    cycler                    0.10.0                   pypi_0    pypi
    databricks-cli            0.14.3                   pypi_0    pypi
    decorator                 5.0.9              pyhd3eb1b0_0
    defusedxml                0.7.1              pyhd3eb1b0_0
    docker                    5.0.0                    pypi_0    pypi
    entrypoints               0.3              py39h06a4308_0
    fcsfiles                  2021.6.6                 pypi_0    pypi
    flask                     2.0.1                    pypi_0    pypi
    flatbuffers               1.12                     pypi_0    pypi
    future                    0.18.2                   pypi_0    pypi
    gast                      0.4.0                    pypi_0    pypi
    gitdb                     4.0.7                    pypi_0    pypi
    gitpython                 3.1.18                   pypi_0    pypi
    google-auth               1.34.0                   pypi_0    pypi
    google-auth-oauthlib      0.4.5                    pypi_0    pypi
    google-pasta              0.2.0                    pypi_0    pypi
    greenlet                  1.1.0                    pypi_0    pypi
    grpcio                    1.34.1                   pypi_0    pypi
    gunicorn                  20.1.0                   pypi_0    pypi
    h5py                      3.1.0                    pypi_0    pypi
    idna                      2.10               pyhd3eb1b0_0
    importlib-metadata        3.10.0           py39h06a4308_0
    importlib_metadata        3.10.0               hd3eb1b0_0
    ipykernel                 5.3.4            py39hb070fc8_0
    ipython                   7.22.0           py39hb070fc8_0
    ipython_genutils          0.2.0              pyhd3eb1b0_1
    itsdangerous              2.0.1                    pypi_0    pypi
    jedi                      0.17.2           py39h06a4308_1
    jinja2                    3.0.1              pyhd3eb1b0_0
    joblib                    1.0.1                    pypi_0    pypi
    json5                     0.9.6              pyhd3eb1b0_0
    jsonschema                3.2.0                      py_2
    jupyter-packaging         0.7.12             pyhd3eb1b0_0
    jupyter_client            6.1.12             pyhd3eb1b0_0
    jupyter_core              4.7.1            py39h06a4308_0
    jupyter_server            1.4.1            py39h06a4308_0
    jupyterlab                3.0.14             pyhd3eb1b0_1
    jupyterlab_pygments       0.1.2                      py_0
    jupyterlab_server         2.6.1              pyhd3eb1b0_0
    keras-nightly             2.5.0.dev2021032900          pypi_0    pypi
    keras-preprocessing       1.1.2                    pypi_0    pypi
    kiwisolver                1.3.1                    pypi_0    pypi
    ld_impl_linux-64          2.35.1               h7274673_9
    libffi                    3.3                  he6710b0_2
    libgcc-ng                 9.3.0               h5101ec6_17
    libgomp                   9.3.0               h5101ec6_17
    libsodium                 1.0.18               h7b6447c_0
    libstdcxx-ng              9.3.0               hd4cf53a_17
    lmfit                     1.0.2                    pypi_0    pypi
    mako                      1.1.4                    pypi_0    pypi
    markdown                  3.3.4                    pypi_0    pypi
    markupsafe                2.0.1            py39h27cfd23_0
    matplotlib                3.4.2                    pypi_0    pypi
    mistune                   0.8.4           py39h27cfd23_1000
    mlflow                    1.19.0                   pypi_0    pypi
    multipletau               0.3.3                    pypi_0    pypi
    nbclassic                 0.2.6              pyhd3eb1b0_0
    nbclient                  0.5.3              pyhd3eb1b0_0
    nbconvert                 6.1.0            py39h06a4308_0
    nbformat                  5.1.3              pyhd3eb1b0_0
    ncurses                   6.2                  he6710b0_1
    nest-asyncio              1.5.1              pyhd3eb1b0_0
    notebook                  6.4.0            py39h06a4308_0
    numpy                     1.19.5                   pypi_0    pypi
    oauthlib                  3.1.1                    pypi_0    pypi
    openssl                   1.1.1k               h27cfd23_0
    opt-einsum                3.3.0                    pypi_0    pypi
    packaging                 21.0               pyhd3eb1b0_0
    pandas                    1.3.1                    pypi_0    pypi
    pandocfilters             1.4.3            py39h06a4308_1
    parso                     0.7.0                      py_0
    pexpect                   4.8.0              pyhd3eb1b0_3
    pickleshare               0.7.5           pyhd3eb1b0_1003
    pillow                    8.3.1                    pypi_0    pypi
    pip                       21.1.3           py39h06a4308_0
    prometheus-flask-exporter 0.18.2                   pypi_0    pypi
    prometheus_client         0.11.0             pyhd3eb1b0_0
    prompt-toolkit            3.0.17             pyh06a4308_0
    protobuf                  3.17.3                   pypi_0    pypi
    ptyprocess                0.7.0              pyhd3eb1b0_2
    pyasn1                    0.4.8                    pypi_0    pypi
    pyasn1-modules            0.2.8                    pypi_0    pypi
    pycparser                 2.20                       py_2
    pygments                  2.9.0              pyhd3eb1b0_0
    pyopenssl                 20.0.1             pyhd3eb1b0_1
    pyparsing                 2.4.7              pyhd3eb1b0_0
    pyrsistent                0.18.0           py39h7f8727e_0
    pysocks                   1.7.1            py39h06a4308_0
    python                    3.9.5                h12debd9_4
    python-dateutil           2.8.2              pyhd3eb1b0_0
    python-editor             1.0.4                    pypi_0    pypi
    pytz                      2021.1             pyhd3eb1b0_0
    pyyaml                    5.4.1                    pypi_0    pypi
    pyzmq                     20.0.0           py39h2531618_1
    querystring-parser        1.2.4                    pypi_0    pypi
    readline                  8.1                  h27cfd23_0
    requests                  2.25.1             pyhd3eb1b0_0
    requests-oauthlib         1.3.0                    pypi_0    pypi
    rsa                       4.7.2                    pypi_0    pypi
    scikit-learn              0.24.2                   pypi_0    pypi
    scipy                     1.7.0                    pypi_0    pypi
    seaborn                   0.11.1                   pypi_0    pypi
    send2trash                1.5.0              pyhd3eb1b0_1
    setuptools                52.0.0           py39h06a4308_0
    six                       1.15.0                   pypi_0    pypi
    smmap                     4.0.0                    pypi_0    pypi
    sniffio                   1.2.0            py39h06a4308_1
    sqlalchemy                1.4.22                   pypi_0    pypi
    sqlite                    3.36.0               hc218d9a_0
    sqlparse                  0.4.1                    pypi_0    pypi
    tabulate                  0.8.9                    pypi_0    pypi
    tensorboard               2.5.0                    pypi_0    pypi
    tensorboard-data-server   0.6.1                    pypi_0    pypi
    tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
    tensorflow                2.5.0                    pypi_0    pypi
    tensorflow-estimator      2.5.0                    pypi_0    pypi
    termcolor                 1.1.0                    pypi_0    pypi
    terminado                 0.9.4            py39h06a4308_0
    testpath                  0.5.0              pyhd3eb1b0_0
    threadpoolctl             2.2.0                    pypi_0    pypi
    tifffile                  2021.7.30                pypi_0    pypi
    tk                        8.6.10               hbc83047_0
    tornado                   6.1              py39h27cfd23_0
    traitlets                 5.0.5              pyhd3eb1b0_0
    typing-extensions         3.7.4.3                  pypi_0    pypi
    tzdata                    2021a                h52ac0ba_0
    uncertainties             3.1.6                    pypi_0    pypi
    urllib3                   1.26.6             pyhd3eb1b0_1
    wcwidth                   0.2.5                      py_0
    webencodings              0.5.1            py39h06a4308_1
    websocket-client          1.1.0                    pypi_0    pypi
    werkzeug                  2.0.1                    pypi_0    pypi
    wheel                     0.36.2             pyhd3eb1b0_0
    wrapt                     1.12.1                   pypi_0    pypi
    xz                        5.2.5                h7b6447c_0
    zeromq                    4.3.4                h2531618_0
    zipp                      3.5.0              pyhd3eb1b0_0
    zlib                      1.2.11               h7b6447c_3
    
    Note: you may need to restart the kernel to use updated packages.
    {'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
     'SLURM_NODELIST': 'node162',
     'SLURM_JOB_NAME': 'bash',
     'XDG_SESSION_ID': '44301',
     'SLURMD_NODENAME': 'node162',
     'SLURM_TOPOLOGY_ADDR': 'node162',
     'SLURM_NTASKS_PER_NODE': '24',
     'HOSTNAME': 'login01',
     'SLURM_PRIO_PROCESS': '0',
     'SLURM_SRUN_COMM_PORT': '34890',
     'SHELL': '/bin/bash',
     'TERM': 'xterm-color',
     'SLURM_JOB_QOS': 'qstand',
     'SLURM_PTY_WIN_ROW': '32',
     'HISTSIZE': '1000',
     'TMPDIR': '/tmp',
     'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
     'SSH_CLIENT': '10.231.181.128 49370 22',
     'CONDA_SHLVL': '2',
     'CONDA_PROMPT_MODIFIER': '(tf) ',
     'QTDIR': '/usr/lib64/qt-3.3',
     'QTINC': '/usr/lib64/qt-3.3/include',
     'SSH_TTY': '/dev/pts/79',
     'NO_PROXY': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001',
     'QT_GRAPHICSSYSTEM_CHECKED': '1',
     'SLURM_NNODES': '1',
     'USER': 'ye53nis',
     'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
     'CONDA_EXE': '/cluster/miniconda3/bin/conda',
     'SLURM_STEP_NUM_NODES': '1',
     'SLURM_JOBID': '1547254',
     'SRUN_DEBUG': '3',
     'FTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'ftp_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_NTASKS': '24',
     'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
     'SLURM_STEP_ID': '0',
     'TMUX': '/tmp/tmux-67339/default,20557,4',
     '_CE_CONDA': '',
     'CONDA_PREFIX_1': '/cluster/miniconda3',
     'SLURM_STEP_LAUNCHER_PORT': '34890',
     'SLURM_TASKS_PER_NODE': '24',
     'MAIL': '/var/spool/mail/ye53nis',
     'PATH': '/home/ye53nis/.conda/envs/tf/bin:/home/lex/Programme/miniconda3/envs/tf-nightly-lab/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
     'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
     'SLURM_JOB_ID': '1547254',
     'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tf',
     'SLURM_JOB_USER': 'ye53nis',
     'SLURM_STEPID': '0',
     'PWD': '/',
     'SLURM_SRUN_COMM_HOST': '192.168.192.5',
     'LANG': 'en_US.UTF-8',
     'SLURM_PTY_WIN_COL': '235',
     'SLURM_UMASK': '0022',
     'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
     'SLURM_JOB_UID': '67339',
     'LOADEDMODULES': '',
     'SLURM_NODEID': '0',
     'TMUX_PANE': '%4',
     'SLURM_SUBMIT_DIR': '/',
     'SLURM_TASK_PID': '37261',
     'SLURM_NPROCS': '24',
     'SLURM_CPUS_ON_NODE': '24',
     'SLURM_DISTRIBUTION': 'block',
     'HTTPS_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'https_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_PROCID': '0',
     'HISTCONTROL': 'ignoredups',
     '_CE_M': '',
     'SLURM_JOB_NODELIST': 'node162',
     'SLURM_PTY_PORT': '39060',
     'HOME': '/home/ye53nis',
     'SHLVL': '3',
     'SLURM_LOCALID': '0',
     'SLURM_JOB_GID': '13280',
     'SLURM_JOB_CPUS_PER_NODE': '24',
     'SLURM_CLUSTER_NAME': 'hpc',
     'no_proxy': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001',
     'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
     'SLURM_SUBMIT_HOST': 'login01',
     'HTTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_JOB_PARTITION': 's_standard',
     'MATHEMATICA_HOME': '/cluster/apps/mathematica/11.3',
     'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
     'LOGNAME': 'ye53nis',
     'SLURM_STEP_NUM_TASKS': '24',
     'QTLIB': '/usr/lib64/qt-3.3/lib',
     'SLURM_JOB_ACCOUNT': 'iaob',
     'SLURM_JOB_NUM_NODES': '1',
     'MODULESHOME': '/usr/share/Modules',
     'CONDA_DEFAULT_ENV': 'tf',
     'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
     'SLURM_STEP_TASKS_PER_NODE': '24',
     'PORT': '9999',
     'SLURM_STEP_NODELIST': 'node162',
     'DISPLAY': ':0',
     'XDG_RUNTIME_DIR': '',
     'XAUTHORITY': '/home/lex/.Xauthority',
     'BASH_FUNC_module()': '() {  eval `/usr/bin/modulecmd bash $*`\n}',
     '_': '/home/ye53nis/.conda/envs/tf/bin/jupyter',
     'JPY_PARENT_PID': '38132',
     'CLICOLOR': '1',
     'PAGER': 'cat',
     'GIT_PAGER': 'cat',
     'MPLBACKEND': 'module://ipykernel.pylab.backend_inline'}
    
2.4.1.3 Node for running Mlflow UI
  1. Create mlflow tmux session and start mlflow ui
    conda activate tf
    mlflow ui --backend-store-uri file:///beegfs/ye53nis/drmed-git/data/mlruns -p 5001
    
    (tf) [ye53nis@login01 ~]$ mlflow ui --backend-store-uri file:///beegfs/ye53nis/drmed-git/data/mlruns -p 5001
    [2021-08-08 14:47:33 +0200] [5106] [INFO] Starting gunicorn 20.1.0
    [2021-08-08 14:47:33 +0200] [5106] [INFO] Listening at: http://127.0.0.1:5001 (5106)
    [2021-08-08 14:47:33 +0200] [5106] [INFO] Using worker: sync
    [2021-08-08 14:47:33 +0200] [5115] [INFO] Booting worker with pid: 5115
    
  2. SHH tunnel the mflow session to the local computer (#+CALL: ssh-tunnel[:session local3](port="5001", node="login01"))
                     
    sh-5.1$ sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:          
    ye53nis@login01's password:              
    bind: Address already in use        
    Last login: Tue Aug 17 18:03:52 2021 from 10.231.188.20

2.4.2 Run 1 - hparams

2.4.2.1 Record metadata
  1. Current directory, last 5 git commits
    pwd
    git log -5
    
    (tf) [ye53nis@node130 drmed-git]$ pwd
    /beegfs/ye53nis/drmed-git
    (tf) [ye53nis@node130 drmed-git]$ git log -5
    commit aa5b9bc35c53c4fd1525c6b812b2a28532ae7afb
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Sat Aug 7 22:15:02 2021 +0200
    
        Add hparams combi restriction; add metadata
    
        problems arise, if in the random combination of hparams, 2*pool_size**n_levels
        is bigger than the input_size. That's why these cases are skipped now.
    
    commit 36bfdd79e78f84fe2f05a11d791c33b9c724b71f
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Sat Aug 7 14:28:28 2021 +0200
    
        add drop_remainder, add hparams, remove steps
    
    commit f3f5310a32a110793d016c2c788b1058bbc5439e
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Sat Aug 7 14:02:57 2021 +0200
    
        Change conda environment
    
    commit fcc80acbc07dcc92b207b034f4b4133f2800d0f3
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Thu Aug 5 22:16:18 2021 +0200
    
        Rename master to main
    
    commit 8edc3c254d26b5c18fb262e73c1fd26406b97573
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Wed Aug 4 22:36:06 2021 +0200
    
        Fix mlflow best run logging
    (tf) [ye53nis@node130 drmed-git]$
    
  2. GPU, CPU, RAM, file system, env variables, top info
    nvcc -V
    echo --------------------
    lscpu
    echo --------------------
    nproc
    echo --------------------
    free -h
    echo --------------------
    df -h
    echo --------------------
    printenv
    echo --------------------
    top -bcn1 -w512 | head -n 15
    
    (tf) [ye53nis@node130 drmed-git]$ nvcc -V
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2020 NVIDIA Corporation
    Built on Mon_Nov_30_19:08:53_PST_2020
    Cuda compilation tools, release 11.2, V11.2.67
    Build cuda_11.2.r11.2/compiler.29373293_0
    
    --------------------
    (tf) [ye53nis@node130 drmed-git]$ lscpu
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                48
    On-line CPU(s) list:   0-47
    Thread(s) per core:    2
    Core(s) per socket:    12
    Socket(s):             2
    NUMA node(s):          4
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 79
    Model name:            Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
    Stepping:              1
    CPU MHz:               1200.439
    CPU max MHz:           2900.0000
    CPU min MHz:           1200.0000
    BogoMIPS:              4399.92
    Virtualization:        VT-x
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              256K
    L3 cache:              15360K
    NUMA node0 CPU(s):     0-5,24-29
    NUMA node1 CPU(s):     6-11,30-35
    NUMA node2 CPU(s):     12-17,36-41
    NUMA node3 CPU(s):     18-23,42-47
    Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arc
    h_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic m
    ovbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_
    adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts spec_ctrl intel_stibp
    
    --------------------
    (tf) [ye53nis@node130 drmed-git]$ nproc
    12
    
    --------------------
    (tf) [ye53nis@node130 drmed-git]$ free -h
                  total        used        free      shared  buff/cache   available
    Mem:           125G        1.2G         78G        229M         45G        122G
    Swap:           11G        2.2M         11G
    
    --------------------
    (tf) [ye53nis@node130 drmed-git]$ df -h
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/sda1             50G  6.2G   44G  13% /
    devtmpfs              63G     0   63G   0% /dev
    tmpfs                 63G  180M   63G   1% /dev/shm
    tmpfs                 63G   51M   63G   1% /run
    tmpfs                 63G     0   63G   0% /sys/fs/cgroup
    nfs03-ib:/pool/work  100T   78T   23T  78% /nfsdata
    nfs01-ib:/home        80T   62T   19T  77% /home
    nfs01-ib:/cluster    2.0T  435G  1.6T  22% /cluster
    /dev/sda6            169G  122M  169G   1% /local
    /dev/sda5            2.0G   35M  2.0G   2% /tmp
    /dev/sda3            6.0G  666M  5.4G  11% /var
    beegfs_nodev         524T  432T   92T  83% /beegfs
    
    --------------------
    (tf) [ye53nis@node130 drmed-git]$ printenv
    SLURM_CHECKPOINT_IMAGE_DIR=/var/slurm/checkpoint
    SLURM_NODELIST=node130
    CUDA_PATH=/cluster/nvidia/cuda/11.2
    SLURM_JOB_NAME=bash
    CUDA_INC_PATH=/cluster/nvidia/cuda/11.2/include
    XDG_SESSION_ID=44301
    SLURMD_NODENAME=node130
    SLURM_TOPOLOGY_ADDR=node130
    SLURM_NTASKS_PER_NODE=12
    HOSTNAME=login01
    SLURM_PRIO_PROCESS=0
    SLURM_SRUN_COMM_PORT=37107
    SHELL=/bin/bash
    TERM=screen
    MLFLOW_EXPERIMENT_NAME=exp-210807-hparams
    SLURM_JOB_QOS=qstand
    SLURM_PTY_WIN_ROW=24
    HISTSIZE=1000
    TMPDIR=/tmp
    SLURM_TOPOLOGY_ADDR_PATTERN=node
    SSH_CLIENT=10.231.181.128 49370 22
    INCLUDEDIR=/cluster/nvidia/cuda/11.2/include
    CONDA_SHLVL=2
    CONDA_PROMPT_MODIFIER=(tf)
    OLDPWD=/beegfs/ye53nis/drmed-git/data
    QTDIR=/usr/lib64/qt-3.3
    QTINC=/usr/lib64/qt-3.3/include
    SSH_TTY=/dev/pts/79
    NO_PROXY=localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001
    QT_GRAPHICSSYSTEM_CHECKED=1
    SLURM_NNODES=1
    USER=ye53nis
    http_proxy=http://internet4nzm.rz.uni-jena.de:3128
    LD_LIBRARY_PATH=/cluster/nvidia/cuda/11.2/lib64:/cluster/nvidia/cuda/11.2/nvvm/lib64:/cluster/nvidia/cudnn/8.1//lib64
    LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01
    ;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01
    ;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01
    ;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tg
    a=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01
    ;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl
    =01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*
    .mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
    CONDA_EXE=/cluster/miniconda3/bin/conda
    SLURM_STEP_NUM_NODES=1
    SLURM_JOBID=1461501
    SRUN_DEBUG=3
    FTP_PROXY=http://internet4nzm.rz.uni-jena.de:3128
    ftp_proxy=http://internet4nzm.rz.uni-jena.de:3128
    SLURM_NTASKS=12
    SLURM_LAUNCH_NODE_IPADDR=192.168.192.5
    SLURM_STEP_ID=0
    TMUX=/tmp/tmux-67339/default,20557,6
    _CE_CONDA=
    CONDA_PREFIX_1=/cluster/miniconda3
    MODCUDA=YES
    SLURM_STEP_LAUNCHER_PORT=37107
    SLURM_TASKS_PER_NODE=12
    MAIL=/var/spool/mail/ye53nis
    PATH=/cluster/nvidia/cuda/11.2/bin:/cluster/nvidia/cuda/11.2/nvvm:/cluster/nvidia/cuda/11.2/open64/bin:/cluster/nvidia/cuda/11.2/libnvvp:/home/ye53nis/.conda/envs/tf/bin:/home/lex/Programm
    e/miniconda3/envs/tf-nightly-lab/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/
    usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin
    SLURM_WORKING_CLUSTER=hpc:192.168.192.1:6817:8448
    SLURM_JOB_ID=1461501
    LD_RUN_PATH=/cluster/nvidia/cuda/11.2/lib64
    SLURM_STEP_GPUS=0
    CONDA_PREFIX=/home/ye53nis/.conda/envs/tf
    CUDA_LIB_PATH=/cluster/nvidia/cuda/11.2/lib64
    SLURM_JOB_USER=ye53nis
    SLURM_STEPID=0
    PWD=/beegfs/ye53nis/drmed-git
    _LMFILES_=/cluster/modulefiles/nvidia/cuda/11.2:/cluster/modulefiles/nvidia/cudnn/8.1
    CUDA_VISIBLE_DEVICES=0
    SLURM_SRUN_COMM_HOST=192.168.192.5
    LANG=en_US.UTF-8
    SLURM_PTY_WIN_COL=80
    SLURM_UMASK=0022
    MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles
    SLURM_JOB_UID=67339
    LOADEDMODULES=nvidia/cuda/11.2:nvidia/cudnn/8.1
    SLURM_NODEID=0
    TMUX_PANE=%6
    SLURM_SUBMIT_DIR=/
    SLURM_TASK_PID=11596
    SLURM_NPROCS=12
    SLURM_CPUS_ON_NODE=12
    SLURM_DISTRIBUTION=block
    HTTPS_PROXY=http://internet4nzm.rz.uni-jena.de:3128
    https_proxy=http://internet4nzm.rz.uni-jena.de:3128
    SLURM_PROCID=0
    HISTCONTROL=ignoredups
    _CE_M=
    SLURM_JOB_NODELIST=node130
    SLURM_PTY_PORT=41921
    HOME=/home/ye53nis
    SHLVL=3
    SLURM_LOCALID=0
    SLURM_JOB_GID=13280
    SLURM_JOB_CPUS_PER_NODE=12
    SLURM_CLUSTER_NAME=hpc
    no_proxy=localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001
    SLURM_GTIDS=0,1,2,3,4,5,6,7,8,9,10,11
    SLURM_SUBMIT_HOST=login01
    HTTP_PROXY=http://internet4nzm.rz.uni-jena.de:3128
    SLURM_JOB_PARTITION=gpu_v100
    MATHEMATICA_HOME=/cluster/apps/mathematica/11.3
    CONDA_PYTHON_EXE=/cluster/miniconda3/bin/python
    LOGNAME=ye53nis
    SLURM_STEP_NUM_TASKS=12
    QTLIB=/usr/lib64/qt-3.3/lib
    GPU_DEVICE_ORDINAL=0
    SLURM_JOB_ACCOUNT=iaob
    MLFLOW_TRACKING_URI=file:/beegfs/ye53nis/drmed-git/data/mlruns
    SLURM_JOB_NUM_NODES=1
    MODULESHOME=/usr/share/Modules
    CONDA_DEFAULT_ENV=tf
    LESSOPEN=||/usr/bin/lesspipe.sh %s
    SLURM_STEP_TASKS_PER_NODE=12
    SLURM_STEP_NODELIST=node130
    DISPLAY=:0
    XDG_RUNTIME_DIR=/run/user/67339
    INCLUDE=/cluster/nvidia/cudnn/8.1//include
    XAUTHORITY=/home/lex/.Xauthority
    BASH_FUNC_module()=() {  eval `/usr/bin/modulecmd bash $*`
    }
    _=/bin/printenv
    
    --------------------
    (tf) [ye53nis@node130 drmed-git]$ top -bcn1 -w512 | head -n 15
    top - 15:07:22 up 89 days, 23:51,  0 users,  load average: 0.00, 0.01, 0.05
    Tasks: 513 total,   2 running, 511 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  0.2 us,  0.2 sy,  0.0 ni, 99.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem : 13191629+total, 82832160 free,  1227056 used, 47857076 buff/cache
    KiB Swap: 12582908 total, 12580604 free,     2304 used. 12863139+avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
     9574 ye53nis   20   0  172600   2616   1668 R  11.8  0.0   0:00.03 top -bcn1 -w512
        1 root      20   0   71816   7552   2584 S   0.0  0.0  38:23.65 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
        2 root      20   0       0      0      0 S   0.0  0.0   0:05.31 [kthreadd]
        3 root      20   0       0      0      0 S   0.0  0.0   2:30.24 [ksoftirqd/0]
        5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 [kworker/0:0H]
        8 root      rt   0       0      0      0 S   0.0  0.0   0:43.26 [migration/0]
        9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 [rcu_bh]
       10 root      20   0       0      0      0 R   0.0  0.0 275:37.95 [rcu_sched]
    (tf) [ye53nis@node130 drmed-git]$
    
  3. print conda list
    conda list
    
    (tf) [ye53nis@node130 drmed-git]$ conda list
    # packages in environment at /home/ye53nis/.conda/envs/tf:
    #
    # Name                    Version                   Build  Channel
    _libgcc_mutex             0.1                        main
    _openmp_mutex             4.5                       1_gnu
    absl-py                   0.13.0                   pypi_0    pypi
    alembic                   1.4.1                    pypi_0    pypi
    anyio                     2.2.0            py39h06a4308_1
    argon2-cffi               20.1.0           py39h27cfd23_1
    asteval                   0.9.25                   pypi_0    pypi
    astunparse                1.6.3                    pypi_0    pypi
    async_generator           1.10               pyhd3eb1b0_0
    attrs                     21.2.0             pyhd3eb1b0_0
    babel                     2.9.1              pyhd3eb1b0_0
    backcall                  0.2.0              pyhd3eb1b0_0
    bleach                    3.3.1              pyhd3eb1b0_0
    brotlipy                  0.7.0           py39h27cfd23_1003
    ca-certificates           2021.7.5             h06a4308_1
    cachetools                4.2.2                    pypi_0    pypi
    certifi                   2021.5.30        py39h06a4308_0
    cffi                      1.14.6           py39h400218f_0
    chardet                   4.0.0           py39h06a4308_1003
    click                     8.0.1                    pypi_0    pypi
    cloudpickle               1.6.0                    pypi_0    pypi
    cryptography              3.4.7            py39hd23ed53_0
    cycler                    0.10.0                   pypi_0    pypi
    databricks-cli            0.14.3                   pypi_0    pypi
    decorator                 5.0.9              pyhd3eb1b0_0
    defusedxml                0.7.1              pyhd3eb1b0_0
    docker                    5.0.0                    pypi_0    pypi
    entrypoints               0.3              py39h06a4308_0
    fcsfiles                  2021.6.6                 pypi_0    pypi
    flask                     2.0.1                    pypi_0    pypi
    flatbuffers               1.12                     pypi_0    pypi
    future                    0.18.2                   pypi_0    pypi
    gast                      0.4.0                    pypi_0    pypi
    gitdb                     4.0.7                    pypi_0    pypi
    gitpython                 3.1.18                   pypi_0    pypi
    google-auth               1.34.0                   pypi_0    pypi
    google-auth-oauthlib      0.4.5                    pypi_0    pypi
    google-pasta              0.2.0                    pypi_0    pypi
    greenlet                  1.1.0                    pypi_0    pypi
    grpcio                    1.34.1                   pypi_0    pypi
    gunicorn                  20.1.0                   pypi_0    pypi
    h5py                      3.1.0                    pypi_0    pypi
    idna                      2.10               pyhd3eb1b0_0
    importlib-metadata        3.10.0           py39h06a4308_0
    importlib_metadata        3.10.0               hd3eb1b0_0
    ipykernel                 5.3.4            py39hb070fc8_0
    ipython                   7.22.0           py39hb070fc8_0
    ipython_genutils          0.2.0              pyhd3eb1b0_1
    itsdangerous              2.0.1                    pypi_0    pypi
    jedi                      0.17.2           py39h06a4308_1
    jinja2                    3.0.1              pyhd3eb1b0_0
    joblib                    1.0.1                    pypi_0    pypi
    json5                     0.9.6              pyhd3eb1b0_0
    jsonschema                3.2.0                      py_2
    jupyter-packaging         0.7.12             pyhd3eb1b0_0
    jupyter_client            6.1.12             pyhd3eb1b0_0
    jupyter_core              4.7.1            py39h06a4308_0
    jupyter_server            1.4.1            py39h06a4308_0
    jupyterlab                3.0.14             pyhd3eb1b0_1
    jupyterlab_pygments       0.1.2                      py_0
    jupyterlab_server         2.6.1              pyhd3eb1b0_0
    keras-nightly             2.5.0.dev2021032900          pypi_0    pypi
    keras-preprocessing       1.1.2                    pypi_0    pypi
    kiwisolver                1.3.1                    pypi_0    pypi
    ld_impl_linux-64          2.35.1               h7274673_9
    libffi                    3.3                  he6710b0_2
    libgcc-ng                 9.3.0               h5101ec6_17
    libgomp                   9.3.0               h5101ec6_17
    libsodium                 1.0.18               h7b6447c_0
    libstdcxx-ng              9.3.0               hd4cf53a_17
    lmfit                     1.0.2                    pypi_0    pypi
    mako                      1.1.4                    pypi_0    pypi
    markdown                  3.3.4                    pypi_0    pypi
    markupsafe                2.0.1            py39h27cfd23_0
    matplotlib                3.4.2                    pypi_0    pypi
    mistune                   0.8.4           py39h27cfd23_1000
    mlflow                    1.19.0                   pypi_0    pypi
    multipletau               0.3.3                    pypi_0    pypi
    nbclassic                 0.2.6              pyhd3eb1b0_0
    nbclient                  0.5.3              pyhd3eb1b0_0
    nbconvert                 6.1.0            py39h06a4308_0
    nbformat                  5.1.3              pyhd3eb1b0_0
    ncurses                   6.2                  he6710b0_1
    nest-asyncio              1.5.1              pyhd3eb1b0_0
    notebook                  6.4.0            py39h06a4308_0
    numpy                     1.19.5                   pypi_0    pypi
    oauthlib                  3.1.1                    pypi_0    pypi
    openssl                   1.1.1k               h27cfd23_0
    opt-einsum                3.3.0                    pypi_0    pypi
    packaging                 21.0               pyhd3eb1b0_0
    pandas                    1.3.1                    pypi_0    pypi
    pandocfilters             1.4.3            py39h06a4308_1
    parso                     0.7.0                      py_0
    pexpect                   4.8.0              pyhd3eb1b0_3
    pickleshare               0.7.5           pyhd3eb1b0_1003
    pillow                    8.3.1                    pypi_0    pypi
    pip                       21.1.3           py39h06a4308_0
    prometheus-flask-exporter 0.18.2                   pypi_0    pypi
    prometheus_client         0.11.0             pyhd3eb1b0_0
    prompt-toolkit            3.0.17             pyh06a4308_0
    protobuf                  3.17.3                   pypi_0    pypi
    ptyprocess                0.7.0              pyhd3eb1b0_2
    pyasn1                    0.4.8                    pypi_0    pypi
    pyasn1-modules            0.2.8                    pypi_0    pypi
    pycparser                 2.20                       py_2
    pygments                  2.9.0              pyhd3eb1b0_0
    pyopenssl                 20.0.1             pyhd3eb1b0_1
    pyparsing                 2.4.7              pyhd3eb1b0_0
    pyrsistent                0.18.0           py39h7f8727e_0
    pysocks                   1.7.1            py39h06a4308_0
    python                    3.9.5                h12debd9_4
    python-dateutil           2.8.2              pyhd3eb1b0_0
    python-editor             1.0.4                    pypi_0    pypi
    pytz                      2021.1             pyhd3eb1b0_0
    pyyaml                    5.4.1                    pypi_0    pypi
    pyzmq                     20.0.0           py39h2531618_1
    querystring-parser        1.2.4                    pypi_0    pypi
    readline                  8.1                  h27cfd23_0
    requests                  2.25.1             pyhd3eb1b0_0
    requests-oauthlib         1.3.0                    pypi_0    pypi
    rsa                       4.7.2                    pypi_0    pypi
    scikit-learn              0.24.2                   pypi_0    pypi
    scipy                     1.7.0                    pypi_0    pypi
    seaborn                   0.11.1                   pypi_0    pypi
    send2trash                1.5.0              pyhd3eb1b0_1
    setuptools                52.0.0           py39h06a4308_0
    six                       1.15.0                   pypi_0    pypi
    smmap                     4.0.0                    pypi_0    pypi
    sniffio                   1.2.0            py39h06a4308_1
    sqlalchemy                1.4.22                   pypi_0    pypi
    sqlite                    3.36.0               hc218d9a_0
    sqlparse                  0.4.1                    pypi_0    pypi
    tabulate                  0.8.9                    pypi_0    pypi
    tensorboard               2.5.0                    pypi_0    pypi
    tensorboard-data-server   0.6.1                    pypi_0    pypi
    tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
    tensorflow                2.5.0                    pypi_0    pypi
    tensorflow-estimator      2.5.0                    pypi_0    pypi
    termcolor                 1.1.0                    pypi_0    pypi
    terminado                 0.9.4            py39h06a4308_0
    testpath                  0.5.0              pyhd3eb1b0_0
    threadpoolctl             2.2.0                    pypi_0    pypi
    tifffile                  2021.7.30                pypi_0    pypi
    tk                        8.6.10               hbc83047_0
    tornado                   6.1              py39h27cfd23_0
    traitlets                 5.0.5              pyhd3eb1b0_0
    typing-extensions         3.7.4.3                  pypi_0    pypi
    tzdata                    2021a                h52ac0ba_0
    uncertainties             3.1.6                    pypi_0    pypi
    urllib3                   1.26.6             pyhd3eb1b0_1
    wcwidth                   0.2.5                      py_0
    webencodings              0.5.1            py39h06a4308_1
    websocket-client          1.1.0                    pypi_0    pypi
    werkzeug                  2.0.1                    pypi_0    pypi
    wheel                     0.36.2             pyhd3eb1b0_0
    wrapt                     1.12.1                   pypi_0    pypi
    xz                        5.2.5                h7b6447c_0
    zeromq                    4.3.4                h2531618_0
    zipp                      3.5.0              pyhd3eb1b0_0
    zlib                      1.2.11               h7b6447c_3
    (tf) [ye53nis@node130 drmed-git]$
    
  4. Show tree of input files used. I made a design choice in simulating these files, which I now regret: I simulated 10 files for each speed of molecules (0.069, 0.08, 0.1, …), but I randomly chose the clusters. In hindsight, the cluster speed creates vastly different artifact shapes, which may have implications for the training. Thats why I manually created different splits of train and val data.
    tree ../saves/firstartifact_Nov2020_train_max3sets
    echo --------------------
    tree ../saves/firstartifact_Nov2020_val_max3sets
    echo --------------------
    tree ../saves/firstartifact_Nov2020_train_max2sets
    echo --------------------
    tree ../saves/firstartifact_Nov2020_val_max2sets_SORTEDIN
    echo --------------------
    tree ../saves/firstartifact_Nov2020_train_max1set
    echo --------------------
    tree ../saves/firstartifact_Nov2020_test
    
    ../saves/firstartifact_Nov2020_train_max3sets
    ├── 0.069
    │   ├── 0.01
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.069_set002.csv
    │   │   ├── traces_brightclust_Nov2020_D0.069_set003.csv
    │   │   └── traces_brightclust_Nov2020_D0.069_set006.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.069_set009.csv
    ├── 0.08
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.08_set007.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.08_set002.csv
    │   │   ├── traces_brightclust_Nov2020_D0.08_set006.csv
    │   │   └── traces_brightclust_Nov2020_D0.08_set008.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.08_set004.csv
    │       └── traces_brightclust_Nov2020_D0.08_set009.csv
    ├── 0.1
    │   ├── 0.01
    │   │   ├── traces_brightclust_Nov2020_D0.1_set004.csv
    │   │   ├── traces_brightclust_Nov2020_D0.1_set006.csv
    │   │   └── traces_brightclust_Nov2020_D0.1_set008.csv
    │   ├── 0.1
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.1_set003.csv
    │       └── traces_brightclust_Nov2020_D0.1_set007.csv
    ├── 0.2
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.2_set003.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.2_set001.csv
    │   │   ├── traces_brightclust_Nov2020_D0.2_set004.csv
    │   │   └── traces_brightclust_Nov2020_D0.2_set006.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.2_set009.csv
    │       └── traces_brightclust_Nov2020_D0.2_set010.csv
    ├── 0.4
    │   ├── 0.01
    │   │   ├── traces_brightclust_Nov2020_D0.4_set004.csv
    │   │   └── traces_brightclust_Nov2020_D0.4_set010.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.4_set002.csv
    │   │   ├── traces_brightclust_Nov2020_D0.4_set003.csv
    │   │   └── traces_brightclust_Nov2020_D0.4_set009.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.4_set006.csv
    │       └── traces_brightclust_Nov2020_D0.4_set007.csv
    ├── 0.6
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.6_set010.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.6_set004.csv
    │   │   ├── traces_brightclust_Nov2020_D0.6_set005.csv
    │   │   └── traces_brightclust_Nov2020_D0.6_set006.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.6_set001.csv
    │       └── traces_brightclust_Nov2020_D0.6_set002.csv
    ├── 10
    │   ├── 0.01
    │   │   ├── traces_brightclust_Nov2020_D10_set003.csv
    │   │   ├── traces_brightclust_Nov2020_D10_set004.csv
    │   │   └── traces_brightclust_Nov2020_D10_set008.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D10_set006.csv
    │   │   └── traces_brightclust_Nov2020_D10_set007.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D10_set010.csv
    ├── 1.0
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D1.0_set010.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D1.0_set004.csv
    │   │   ├── traces_brightclust_Nov2020_D1.0_set007.csv
    │   │   └── traces_brightclust_Nov2020_D1.0_set009.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D1.0_set001.csv
    │       ├── traces_brightclust_Nov2020_D1.0_set002.csv
    │       └── traces_brightclust_Nov2020_D1.0_set008.csv
    ├── 3.0
    │   ├── 0.01
    │   │   ├── traces_brightclust_Nov2020_D3.0_set005.csv
    │   │   ├── traces_brightclust_Nov2020_D3.0_set006.csv
    │   │   └── traces_brightclust_Nov2020_D3.0_set008.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D3.0_set010.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D3.0_set001.csv
    │       ├── traces_brightclust_Nov2020_D3.0_set003.csv
    │       └── traces_brightclust_Nov2020_D3.0_set009.csv
    └── 50
        ├── 0.01
        │   └── traces_brightclust_Nov2020_D50_set006.csv
        ├── 0.1
        │   ├── traces_brightclust_Nov2020_D50_set009.csv
        │   └── traces_brightclust_Nov2020_D50_set010.csv
        └── 1.0
            ├── traces_brightclust_Nov2020_D50_set004.csv
            ├── traces_brightclust_Nov2020_D50_set005.csv
            └── traces_brightclust_Nov2020_D50_set007.csv
    
    40 directories, 60 files
    
    --------------------
    ../saves/firstartifact_Nov2020_val_max3sets
    ├── 0.069
    │   ├── 0.01
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.069_set007.csv
    │   │   └── traces_brightclust_Nov2020_D0.069_set008.csv
    │   └── 1.0
    ├── 0.08
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.08_set010.csv
    │   └── 1.0
    ├── 0.1
    │   ├── 0.01
    │   │   ├── traces_brightclust_Nov2020_D0.1_set009.csv
    │   │   └── traces_brightclust_Nov2020_D0.1_set010.csv
    │   ├── 0.1
    │   └── 1.0
    ├── 0.2
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.2_set008.csv
    │   └── 1.0
    ├── 0.4
    │   ├── 0.01
    │   ├── 0.1
    │   └── 1.0
    ├── 0.6
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.6_set007.csv
    │   └── 1.0
    ├── 10
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D10_set009.csv
    │   ├── 0.1
    │   └── 1.0
    ├── 1.0
    │   ├── 0.01
    │   ├── 0.1
    │   └── 1.0
    ├── 3.0
    │   ├── 0.01
    │   ├── 0.1
    │   └── 1.0
    └── 50
        ├── 0.01
        ├── 0.1
        └── 1.0
            └── traces_brightclust_Nov2020_D50_set008.csv
    
    40 directories, 9 files
    
    --------------------
    ../saves/firstartifact_Nov2020_train_max2sets
    ├── 0.069
    │   ├── 0.01
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.069_set002.csv
    │   │   └── traces_brightclust_Nov2020_D0.069_set003.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.069_set009.csv
    ├── 0.08
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.08_set007.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.08_set002.csv
    │   │   └── traces_brightclust_Nov2020_D0.08_set006.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.08_set004.csv
    │       └── traces_brightclust_Nov2020_D0.08_set009.csv
    ├── 0.1
    │   ├── 0.01
    │   │   ├── traces_brightclust_Nov2020_D0.1_set004.csv
    │   │   └── traces_brightclust_Nov2020_D0.1_set006.csv
    │   ├── 0.1
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.1_set003.csv
    │       └── traces_brightclust_Nov2020_D0.1_set007.csv
    ├── 0.2
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.2_set003.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.2_set001.csv
    │   │   └── traces_brightclust_Nov2020_D0.2_set004.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.2_set009.csv
    │       └── traces_brightclust_Nov2020_D0.2_set010.csv
    ├── 0.4
    │   ├── 0.01
    │   │   ├── traces_brightclust_Nov2020_D0.4_set004.csv
    │   │   └── traces_brightclust_Nov2020_D0.4_set010.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.4_set002.csv
    │   │   └── traces_brightclust_Nov2020_D0.4_set003.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.4_set006.csv
    │       └── traces_brightclust_Nov2020_D0.4_set007.csv
    ├── 0.6
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.6_set010.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.6_set004.csv
    │   │   └── traces_brightclust_Nov2020_D0.6_set005.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.6_set001.csv
    │       └── traces_brightclust_Nov2020_D0.6_set002.csv
    ├── 10
    │   ├── 0.01
    │   │   ├── traces_brightclust_Nov2020_D10_set003.csv
    │   │   └── traces_brightclust_Nov2020_D10_set004.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D10_set006.csv
    │   │   └── traces_brightclust_Nov2020_D10_set007.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D10_set010.csv
    ├── 1.0
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D1.0_set010.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D1.0_set004.csv
    │   │   └── traces_brightclust_Nov2020_D1.0_set007.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D1.0_set001.csv
    │       └── traces_brightclust_Nov2020_D1.0_set002.csv
    ├── 3.0
    │   ├── 0.01
    │   │   ├── traces_brightclust_Nov2020_D3.0_set005.csv
    │   │   └── traces_brightclust_Nov2020_D3.0_set006.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D3.0_set010.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D3.0_set001.csv
    │       └── traces_brightclust_Nov2020_D3.0_set003.csv
    └── 50
        ├── 0.01
        │   └── traces_brightclust_Nov2020_D50_set006.csv
        ├── 0.1
        │   ├── traces_brightclust_Nov2020_D50_set009.csv
        │   └── traces_brightclust_Nov2020_D50_set010.csv
        └── 1.0
            ├── traces_brightclust_Nov2020_D50_set004.csv
            └── traces_brightclust_Nov2020_D50_set005.csv
    
    40 directories, 48 files
    
    --------------------
    ../saves/firstartifact_Nov2020_val_max2sets_SORTEDIN
    ├── 0.069
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.069_set006.csv
    │   └── 1.0
    ├── 0.08
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.08_set008.csv
    │   └── 1.0
    ├── 0.1
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.1_set008.csv
    │   ├── 0.1
    │   └── 1.0
    ├── 0.2
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.2_set006.csv
    │   └── 1.0
    ├── 0.4
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.4_set009.csv
    │   └── 1.0
    ├── 0.6
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.6_set006.csv
    │   └── 1.0
    ├── 10
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D10_set008.csv
    │   ├── 0.1
    │   └── 1.0
    ├── 1.0
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D1.0_set009.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D1.0_set008.csv
    ├── 3.0
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D3.0_set008.csv
    │   ├── 0.1
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D3.0_set009.csv
    └── 50
        ├── 0.01
        ├── 0.1
        └── 1.0
            └── traces_brightclust_Nov2020_D50_set007.csv
    
    40 directories, 12 files
    
    --------------------
    ../saves/firstartifact_Nov2020_train_max1set
    ├── 0.069
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.069_set002.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.069_set009.csv
    ├── 0.08
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.08_set007.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.08_set002.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.08_set004.csv
    ├── 0.1
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.1_set004.csv
    │   ├── 0.1
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.1_set003.csv
    ├── 0.2
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.2_set003.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.2_set001.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.2_set009.csv
    ├── 0.4
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.4_set004.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.4_set002.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.4_set006.csv
    ├── 0.6
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.6_set010.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.6_set004.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.6_set001.csv
    ├── 10
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D10_set003.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D10_set006.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D10_set010.csv
    ├── 1.0
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D1.0_set010.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D1.0_set004.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D1.0_set001.csv
    ├── 3.0
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D3.0_set005.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D3.0_set010.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D3.0_set001.csv
    └── 50
        ├── 0.01
        │   └── traces_brightclust_Nov2020_D50_set006.csv
        ├── 0.1
        │   └── traces_brightclust_Nov2020_D50_set009.csv
        └── 1.0
            └── traces_brightclust_Nov2020_D50_set004.csv
    
    39 directories, 28 files
    
    --------------------
    ../saves/firstartifact_Nov2020_test
    ├── 0.069
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.069_set005.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.069_set001.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.069_set010.csv
    ├── 0.08
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.08_set005.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.08_set003.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.08_set001.csv
    ├── 0.1
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.1_set002.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.1_set005.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.1_set001.csv
    ├── 0.2
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.2_set002.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.2_set007.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.2_set005.csv
    ├── 0.4
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.4_set008.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.4_set001.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.4_set005.csv
    ├── 0.6
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.6_set008.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.6_set003.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.6_set009.csv
    ├── 10
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D10_set002.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D10_set001.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D10_set005.csv
    ├── 1.0
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D1.0_set006.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D1.0_set003.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D1.0_set005.csv
    ├── 3.0
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D3.0_set004.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D3.0_set007.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D3.0_set002.csv
    └── 50
        ├── 0.01
        │   └── traces_brightclust_Nov2020_D50_set002.csv
        ├── 0.1
        │   └── traces_brightclust_Nov2020_D50_set003.csv
        └── 1.0
            └── traces_brightclust_Nov2020_D50_set001.csv
    
    40 directories, 30 files
    
    
2.4.2.2 Mlflow run 1 (failed mid run)
  • Use firstartifact_Nov2020_train_max2sets and firstartifact_Nov2020_val_max2sets_SORTEDIN to get a good equilibrium between amount of training data (max1set would maybe be not enough) and equal distribution of artifacts (max3sets might give too much weight to slow clusters with 0.1 speed, because a lot of them were simulated). Note that the #+RESULTS section was too long, that's why not everything is printed (but logged in MLFLOW of course)
    mlflow run . -e search_hparams -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets -P csv_path_val=/beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN -P num_session_groups=70
    
    0000 - val_precision0.7: 0.1635 - val_recall0.7: 0.9999 - val_tp0.9: 803718.0000 - val_fp0.9: 4111163.0000 - val_tn0.9: 180.0000 - val_fn0.9: 139.0000 - val_precision0.9: 0.1635 - val_reca
    ll0.9: 0.9998 - val_accuracy: 0.1635 - val_auc: 0.5000 - val_f1: 0.2811
    Epoch 6/20
    2400/2400 [==============================] - 63s 26ms/step - loss: 0.5782 - tp0.1: 2622910.0000 - fp0.1: 2505997.0000 - tn0.1: 14176971.0000 - fn0.1: 354922.0000 - precision0.1: 0.5114 - r
    ecall0.1: 0.8808 - tp0.3: 2342499.0000 - fp0.3: 1169993.0000 - tn0.3: 15512975.0000 - fn0.3: 635333.0000 - precision0.3: 0.6669 - recall0.3: 0.7866 - tp0.5: 2053203.0000 - fp0.5: 564864.00
    00 - tn0.5: 16118104.0000 - fn0.5: 924629.0000 - precision0.5: 0.7842 - recall0.5: 0.6895 - tp0.7: 1755923.0000 - fp0.7: 242889.0000 - tn0.7: 16440079.0000 - fn0.7: 1221909.0000 - precisio
    n0.7: 0.8785 - recall0.7: 0.5897 - tp0.9: 1327740.0000 - fp0.9: 66010.0000 - tn0.9: 16616958.0000 - fn0.9: 1650092.0000 - precision0.9: 0.9526 - recall0.9: 0.4459 - accuracy: 0.9242 - auc:
     0.9204 - f1: 0.2631 - val_loss: 13.4848 - val_tp0.1: 750399.0000 - val_fp0.1: 3679523.0000 - val_tn0.1: 431820.0000 - val_fn0.1: 53458.0000 - val_precision0.1: 0.1694 - val_recall0.1: 0.9
    335 - val_tp0.3: 743640.0000 - val_fp0.3: 3633172.0000 - val_tn0.3: 478171.0000 - val_fn0.3: 60217.0000 - val_precision0.3: 0.1699 - val_recall0.3: 0.9251 - val_tp0.5: 739322.0000 - val_fp
    0.5: 3600015.0000 - val_tn0.5: 511328.0000 - val_fn0.5: 64535.0000 - val_precision0.5: 0.1704 - val_recall0.5: 0.9197 - val_tp0.7: 734726.0000 - val_fp0.7: 3563811.0000 - val_tn0.7: 547532
    .0000 - val_fn0.7: 69131.0000 - val_precision0.7: 0.1709 - val_recall0.7: 0.9140 - val_tp0.9: 727004.0000 - val_fp0.9: 3500250.0000 - val_tn0.9: 611093.0000 - val_fn0.9: 76853.0000 - val_p
    recision0.9: 0.1720 - val_recall0.9: 0.9044 - val_accuracy: 0.2544 - val_auc: 0.5331 - val_f1: 0.2811
    Epoch 7/20
    2400/2400 [==============================] - 63s 26ms/step - loss: 0.5493 - tp0.1: 2634892.0000 - fp0.1: 2322719.0000 - tn0.1: 14360249.0000 - fn0.1: 342940.0000 - precision0.1: 0.5315 - r
    ecall0.1: 0.8848 - tp0.3: 2368274.0000 - fp0.3: 1081444.0000 - tn0.3: 15601524.0000 - fn0.3: 609558.0000 - precision0.3: 0.6865 - recall0.3: 0.7953 - tp0.5: 2112193.0000 - fp0.5: 526093.00
    00 - tn0.5: 16156875.0000 - fn0.5: 865639.0000 - precision0.5: 0.8006 - recall0.5: 0.7093 - tp0.7: 1831510.0000 - fp0.7: 239952.0000 - tn0.7: 16443016.0000 - fn0.7: 1146322.0000 - precisio
    n0.7: 0.8842 - recall0.7: 0.6150 - tp0.9: 1405225.0000 - fp0.9: 69139.0000 - tn0.9: 16613829.0000 - fn0.9: 1572607.0000 - precision0.9: 0.9531 - recall0.9: 0.4719 - accuracy: 0.9292 - auc:
     0.9254 - f1: 0.2631 - val_loss: 7.7748 - val_tp0.1: 803044.0000 - val_fp0.1: 4110345.0000 - val_tn0.1: 998.0000 - val_fn0.1: 813.0000 - val_precision0.1: 0.1634 - val_recall0.1: 0.9990 -
    val_tp0.3: 800982.0000 - val_fp0.3: 4110344.0000 - val_tn0.3: 999.0000 - val_fn0.3: 2875.0000 - val_precision0.3: 0.1631 - val_recall0.3: 0.9964 - val_tp0.5: 797547.0000 - val_fp0.5: 41103
    41.0000 - val_tn0.5: 1002.0000 - val_fn0.5: 6310.0000 - val_precision0.5: 0.1625 - val_recall0.5: 0.9922 - val_tp0.7: 790114.0000 - val_fp0.7: 4110336.0000 - val_tn0.7: 1007.0000 - val_fn0
    .7: 13743.0000 - val_precision0.7: 0.1612 - val_recall0.7: 0.9829 - val_tp0.9: 762283.0000 - val_fp0.9: 4110237.0000 - val_tn0.9: 1106.0000 - val_fn0.9: 41574.0000 - val_precision0.9: 0.15
    64 - val_recall0.9: 0.9483 - val_accuracy: 0.1625 - val_auc: 0.4107 - val_f1: 0.2811
    Epoch 8/20
    2400/2400 [==============================] - 63s 26ms/step - loss: 0.5317 - tp0.1: 2659295.0000 - fp0.1: 2333208.0000 - tn0.1: 14349760.0000 - fn0.1: 318537.0000 - precision0.1: 0.5327 - r
    ecall0.1: 0.8930 - tp0.3: 2408124.0000 - fp0.3: 1118900.0000 - tn0.3: 15564068.0000 - fn0.3: 569708.0000 - precision0.3: 0.6828 - recall0.3: 0.8087 - tp0.5: 2142570.0000 - fp0.5: 537358.00
    00 - tn0.5: 16145610.0000 - fn0.5: 835262.0000 - precision0.5: 0.7995 - recall0.5: 0.7195 - tp0.7: 1851259.0000 - fp0.7: 238215.0000 - tn0.7: 16444753.0000 - fn0.7: 1126573.0000 - precisio
    n0.7: 0.8860 - recall0.7: 0.6217 - tp0.9: 1401657.0000 - fp0.9: 63157.0000 - tn0.9: 16619811.0000 - fn0.9: 1576175.0000 - precision0.9: 0.9569 - recall0.9: 0.4707 - accuracy: 0.9302 - auc:
     0.9298 - f1: 0.2631 - val_loss: 3.0251 - val_tp0.1: 49509.0000 - val_fp0.1: 176370.0000 - val_tn0.1: 3934973.0000 - val_fn0.1: 754348.0000 - val_precision0.1: 0.2192 - val_recall0.1: 0.06
    16 - val_tp0.3: 41217.0000 - val_fp0.3: 153934.0000 - val_tn0.3: 3957409.0000 - val_fn0.3: 762640.0000 - val_precision0.3: 0.2112 - val_recall0.3: 0.0513 - val_tp0.5: 36870.0000 - val_fp0.
    5: 140528.0000 - val_tn0.5: 3970815.0000 - val_fn0.5: 766987.0000 - val_precision0.5: 0.2078 - val_recall0.5: 0.0459 - val_tp0.7: 32949.0000 - val_fp0.7: 127605.0000 - val_tn0.7: 3983738.0
    000 - val_fn0.7: 770908.0000 - val_precision0.7: 0.2052 - val_recall0.7: 0.0410 - val_tp0.9: 27690.0000 - val_fp0.9: 108151.0000 - val_tn0.9: 4003192.0000 - val_fn0.9: 776167.0000 - val_pr
    ecision0.9: 0.2038 - val_recall0.9: 0.0344 - val_accuracy: 0.8154 - val_auc: 0.5157 - val_f1: 0.2811
    Epoch 9/20
    2400/2400 [==============================] - 63s 26ms/step - loss: 0.5086 - tp0.1: 2674117.0000 - fp0.1: 2256419.0000 - tn0.1: 14426549.0000 - fn0.1: 303715.0000 - precision0.1: 0.5424 - r
    ecall0.1: 0.8980 - tp0.3: 2438091.0000 - fp0.3: 1100372.0000 - tn0.3: 15582596.0000 - fn0.3: 539741.0000 - precision0.3: 0.6890 - recall0.3: 0.8187 - tp0.5: 2186313.0000 - fp0.5: 529438.00
    00 - tn0.5: 16153530.0000 - fn0.5: 791519.0000 - precision0.5: 0.8050 - recall0.5: 0.7342 - tp0.7: 1892609.0000 - fp0.7: 231011.0000 - tn0.7: 16451957.0000 - fn0.7: 1085223.0000 - precisio
    n0.7: 0.8912 - recall0.7: 0.6356 - tp0.9: 1451523.0000 - fp0.9: 61651.0000 - tn0.9: 16621317.0000 - fn0.9: 1526309.0000 - precision0.9: 0.9593 - recall0.9: 0.4874 - accuracy: 0.9328 - auc:
     0.9337 - f1: 0.2631 - val_loss: 2.5628 - val_tp0.1: 369859.0000 - val_fp0.1: 118128.0000 - val_tn0.1: 3993215.0000 - val_fn0.1: 433998.0000 - val_precision0.1: 0.7579 - val_recall0.1: 0.4
    601 - val_tp0.3: 356259.0000 - val_fp0.3: 94845.0000 - val_tn0.3: 4016498.0000 - val_fn0.3: 447598.0000 - val_precision0.3: 0.7897 - val_recall0.3: 0.4432 - val_tp0.5: 347560.0000 - val_fp
    0.5: 82338.0000 - val_tn0.5: 4029005.0000 - val_fn0.5: 456297.0000 - val_precision0.5: 0.8085 - val_recall0.5: 0.4324 - val_tp0.7: 338552.0000 - val_fp0.7: 70840.0000 - val_tn0.7: 4040503.
    0000 - val_fn0.7: 465305.0000 - val_precision0.7: 0.8270 - val_recall0.7: 0.4212 - val_tp0.9: 317996.0000 - val_fp0.9: 52787.0000 - val_tn0.9: 4058556.0000 - val_fn0.9: 485861.0000 - val_p
    recision0.9: 0.8576 - val_recall0.9: 0.3956 - val_accuracy: 0.8904 - val_auc: 0.7315 - val_f1: 0.2812
    Epoch 10/20
    2400/2400 [==============================] - 63s 26ms/step - loss: 0.4747 - tp0.1: 2701201.0000 - fp0.1: 2158402.0000 - tn0.1: 14524566.0000 - fn0.1: 276631.0000 - precision0.1: 0.5558 - r
    ecall0.1: 0.9071 - tp0.3: 2471097.0000 - fp0.3: 1011755.0000 - tn0.3: 15671213.0000 - fn0.3: 506735.0000 - precision0.3: 0.7095 - recall0.3: 0.8298 - tp0.5: 2234639.0000 - fp0.5: 500413.00
    00 - tn0.5: 16182555.0000 - fn0.5: 743193.0000 - precision0.5: 0.8170 - recall0.5: 0.7504 - tp0.7: 1960526.0000 - fp0.7: 224293.0000 - tn0.7: 16458675.0000 - fn0.7: 1017306.0000 - precisio
    n0.7: 0.8973 - recall0.7: 0.6584 - tp0.9: 1515400.0000 - fp0.9: 58089.0000 - tn0.9: 16624879.0000 - fn0.9: 1462432.0000 - precision0.9: 0.9631 - recall0.9: 0.5089 - accuracy: 0.9367 - auc:
     0.9403 - f1: 0.2631 - val_loss: 2.7288 - val_tp0.1: 770447.0000 - val_fp0.1: 3861936.0000 - val_tn0.1: 249407.0000 - val_fn0.1: 33410.0000 - val_precision0.1: 0.1663 - val_recall0.1: 0.95
    84 - val_tp0.3: 749545.0000 - val_fp0.3: 3747496.0000 - val_tn0.3: 363847.0000 - val_fn0.3: 54312.0000 - val_precision0.3: 0.1667 - val_recall0.3: 0.9324 - val_tp0.5: 720064.0000 - val_fp0
    .5: 3582604.0000 - val_tn0.5: 528739.0000 - val_fn0.5: 83793.0000 - val_precision0.5: 0.1674 - val_recall0.5: 0.8958 - val_tp0.7: 648811.0000 - val_fp0.7: 3077696.0000 - val_tn0.7: 1033647
    .0000 - val_fn0.7: 155046.0000 - val_precision0.7: 0.1741 - val_recall0.7: 0.8071 - val_tp0.9: 426993.0000 - val_fp0.9: 1072419.0000 - val_tn0.9: 3038924.0000 - val_fn0.9: 376864.0000 - va
    l_precision0.9: 0.2848 - val_recall0.9: 0.5312 - val_accuracy: 0.2541 - val_auc: 0.6440 - val_f1: 0.2811
    Epoch 11/20
    2400/2400 [==============================] - 63s 26ms/step - loss: 0.4648 - tp0.1: 2705161.0000 - fp0.1: 2061104.0000 - tn0.1: 14621864.0000 - fn0.1: 272671.0000 - precision0.1: 0.5676 - r
    ecall0.1: 0.9084 - tp0.3: 2495871.0000 - fp0.3: 1013446.0000 - tn0.3: 15669522.0000 - fn0.3: 481961.0000 - precision0.3: 0.7112 - recall0.3: 0.8382 - tp0.5: 2264736.0000 - fp0.5: 501786.00
    00 - tn0.5: 16181182.0000 - fn0.5: 713096.0000 - precision0.5: 0.8186 - recall0.5: 0.7605 - tp0.7: 1995279.0000 - fp0.7: 225179.0000 - tn0.7: 16457789.0000 - fn0.7: 982553.0000 - precision
    0.7: 0.8986 - recall0.7: 0.6700 - tp0.9: 1553742.0000 - fp0.9: 58202.0000 - tn0.9: 16624766.0000 - fn0.9: 1424090.0000 - precision0.9: 0.9639 - recall0.9: 0.5218 - accuracy: 0.9382 - auc:
    0.9416 - f1: 0.2631 - val_loss: 2.5613 - val_tp0.1: 412390.0000 - val_fp0.1: 2079926.0000 - val_tn0.1: 2031417.0000 - val_fn0.1: 391467.0000 - val_precision0.1: 0.1655 - val_recall0.1: 0.5
    130 - val_tp0.3: 404704.0000 - val_fp0.3: 2059044.0000 - val_tn0.3: 2052299.0000 - val_fn0.3: 399153.0000 - val_precision0.3: 0.1643 - val_recall0.3: 0.5035 - val_tp0.5: 396367.0000 - val_
    fp0.5: 2012051.0000 - val_tn0.5: 2099292.0000 - val_fn0.5: 407490.0000 - val_precision0.5: 0.1646 - val_recall0.5: 0.4931 - val_tp0.7: 362322.0000 - val_fp0.7: 1698034.0000 - val_tn0.7: 24
    13309.0000 - val_fn0.7: 441535.0000 - val_precision0.7: 0.1759 - val_recall0.7: 0.4507 - val_tp0.9: 186125.0000 - val_fp0.9: 532408.0000 - val_tn0.9: 3578935.0000 - val_fn0.9: 617732.0000
    - val_precision0.9: 0.2590 - val_recall0.9: 0.2315 - val_accuracy: 0.5077 - val_auc: 0.5355 - val_f1: 0.2811
    Epoch 12/20
    2400/2400 [==============================] - 64s 26ms/step - loss: 0.4441 - tp0.1: 2719890.0000 - fp0.1: 2017765.0000 - tn0.1: 14665203.0000 - fn0.1: 257942.0000 - precision0.1: 0.5741 - r
    ecall0.1: 0.9134 - tp0.3: 2513777.0000 - fp0.3: 998230.0000 - tn0.3: 15684738.0000 - fn0.3: 464055.0000 - precision0.3: 0.7158 - recall0.3: 0.8442 - tp0.5: 2283999.0000 - fp0.5: 506817.000
    0 - tn0.5: 16176151.0000 - fn0.5: 693833.0000 - precision0.5: 0.8184 - recall0.5: 0.7670 - tp0.7: 2013621.0000 - fp0.7: 228901.0000 - tn0.7: 16454067.0000 - fn0.7: 964211.0000 - precision0
    .7: 0.8979 - recall0.7: 0.6762 - tp0.9: 1553339.0000 - fp0.9: 58704.0000 - tn0.9: 16624264.0000 - fn0.9: 1424493.0000 - precision0.9: 0.9636 - recall0.9: 0.5216 - accuracy: 0.9389 - auc: 0
    .9450 - f1: 0.2631 - val_loss: 7.1302 - val_tp0.1: 605571.0000 - val_fp0.1: 2024528.0000 - val_tn0.1: 2086815.0000 - val_fn0.1: 198286.0000 - val_precision0.1: 0.2302 - val_recall0.1: 0.75
    33 - val_tp0.3: 600395.0000 - val_fp0.3: 1954684.0000 - val_tn0.3: 2156659.0000 - val_fn0.3: 203462.0000 - val_precision0.3: 0.2350 - val_recall0.3: 0.7469 - val_tp0.5: 597139.0000 - val_f
    p0.5: 1912183.0000 - val_tn0.5: 2199160.0000 - val_fn0.5: 206718.0000 - val_precision0.5: 0.2380 - val_recall0.5: 0.7428 - val_tp0.7: 594024.0000 - val_fp0.7: 1870774.0000 - val_tn0.7: 224
    0569.0000 - val_fn0.7: 209833.0000 - val_precision0.7: 0.2410 - val_recall0.7: 0.7390 - val_tp0.9: 589086.0000 - val_fp0.9: 1806611.0000 - val_tn0.9: 2304732.0000 - val_fn0.9: 214771.0000
    - val_precision0.9: 0.2459 - val_recall0.9: 0.7328 - val_accuracy: 0.5689 - val_auc: 0.6516 - val_f1: 0.2811
    Epoch 13/20
    2400/2400 [==============================] - 63s 26ms/step - loss: 0.4185 - tp0.1: 2737060.0000 - fp0.1: 1927283.0000 - tn0.1: 14755685.0000 - fn0.1: 240772.0000 - precision0.1: 0.5868 - r
    ecall0.1: 0.9191 - tp0.3: 2543087.0000 - fp0.3: 952973.0000 - tn0.3: 15729995.0000 - fn0.3: 434745.0000 - precision0.3: 0.7274 - recall0.3: 0.8540 - tp0.5: 2330175.0000 - fp0.5: 482868.000
    0 - tn0.5: 16200100.0000 - fn0.5: 647657.0000 - precision0.5: 0.8283 - recall0.5: 0.7825 - tp0.7: 2076869.0000 - fp0.7: 221622.0000 - tn0.7: 16461346.0000 - fn0.7: 900963.0000 - precision0
    .7: 0.9036 - recall0.7: 0.6974 - tp0.9: 1634082.0000 - fp0.9: 57693.0000 - tn0.9: 16625275.0000 - fn0.9: 1343750.0000 - precision0.9: 0.9659 - recall0.9: 0.5487 - accuracy: 0.9425 - auc: 0
    .9493 - f1: 0.2631 - val_loss: 1.8139 - val_tp0.1: 239760.0000 - val_fp0.1: 4548.0000 - val_tn0.1: 4106795.0000 - val_fn0.1: 564097.0000 - val_precision0.1: 0.9814 - val_recall0.1: 0.2983
    - val_tp0.3: 233628.0000 - val_fp0.3: 3848.0000 - val_tn0.3: 4107495.0000 - val_fn0.3: 570229.0000 - val_precision0.3: 0.9838 - val_recall0.3: 0.2906 - val_tp0.5: 227509.0000 - val_fp0.5:
    3252.0000 - val_tn0.5: 4108091.0000 - val_fn0.5: 576348.0000 - val_precision0.5: 0.9859 - val_recall0.5: 0.2830 - val_tp0.7: 217711.0000 - val_fp0.7: 2455.0000 - val_tn0.7: 4108888.0000 -
    val_fn0.7: 586146.0000 - val_precision0.7: 0.9888 - val_recall0.7: 0.2708 - val_tp0.9: 195925.0000 - val_fp0.9: 1407.0000 - val_tn0.9: 4109936.0000 - val_fn0.9: 607932.0000 - val_precision
    0.9: 0.9929 - val_recall0.9: 0.2437 - val_accuracy: 0.8821 - val_auc: 0.6547 - val_f1: 0.2811
    Epoch 14/20
    2400/2400 [==============================] - 63s 26ms/step - loss: 0.4189 - tp0.1: 2744214.0000 - fp0.1: 1995913.0000 - tn0.1: 14687055.0000 - fn0.1: 233618.0000 - precision0.1: 0.5789 - r
    ecall0.1: 0.9215 - tp0.3: 2544539.0000 - fp0.3: 941608.0000 - tn0.3: 15741360.0000 - fn0.3: 433293.0000 - precision0.3: 0.7299 - recall0.3: 0.8545 - tp0.5: 2330855.0000 - fp0.5: 475880.000
    0 - tn0.5: 16207088.0000 - fn0.5: 646977.0000 - precision0.5: 0.8305 - recall0.5: 0.7827 - tp0.7: 2079586.0000 - fp0.7: 222500.0000 - tn0.7: 16460468.0000 - fn0.7: 898246.0000 - precision0
    .7: 0.9033 - recall0.7: 0.6984 - tp0.9: 1628422.0000 - fp0.9: 55899.0000 - tn0.9: 16627069.0000 - fn0.9: 1349410.0000 - precision0.9: 0.9668 - recall0.9: 0.5468 - accuracy: 0.9429 - auc: 0
    .9503 - f1: 0.2631 - val_loss: 1.6218 - val_tp0.1: 510355.0000 - val_fp0.1: 1689260.0000 - val_tn0.1: 2422083.0000 - val_fn0.1: 293502.0000 - val_precision0.1: 0.2320 - val_recall0.1: 0.63
    49 - val_tp0.3: 421290.0000 - val_fp0.3: 1170939.0000 - val_tn0.3: 2940404.0000 - val_fn0.3: 382567.0000 - val_precision0.3: 0.2646 - val_recall0.3: 0.5241 - val_tp0.5: 358225.0000 - val_f
    p0.5: 571750.0000 - val_tn0.5: 3539593.0000 - val_fn0.5: 445632.0000 - val_precision0.5: 0.3852 - val_recall0.5: 0.4456 - val_tp0.7: 323677.0000 - val_fp0.7: 214942.0000 - val_tn0.7: 38964
    01.0000 - val_fn0.7: 480180.0000 - val_precision0.7: 0.6009 - val_recall0.7: 0.4027 - val_tp0.9: 290885.0000 - val_fp0.9: 143266.0000 - val_tn0.9: 3968077.0000 - val_fn0.9: 512972.0000 - v
    al_precision0.9: 0.6700 - val_recall0.9: 0.3619 - val_accuracy: 0.7930 - val_auc: 0.6785 - val_f1: 0.2811
    Epoch 15/20
    2400/2400 [==============================] - 63s 26ms/step - loss: 0.4039 - tp0.1: 2751584.0000 - fp0.1: 1904870.0000 - tn0.1: 14778098.0000 - fn0.1: 226248.0000 - precision0.1: 0.5909 - r
    ecall0.1: 0.9240 - tp0.3: 2559610.0000 - fp0.3: 920199.0000 - tn0.3: 15762769.0000 - fn0.3: 418222.0000 - precision0.3: 0.7356 - recall0.3: 0.8596 - tp0.5: 2358628.0000 - fp0.5: 480866.000
    0 - tn0.5: 16202102.0000 - fn0.5: 619204.0000 - precision0.5: 0.8307 - recall0.5: 0.7921 - tp0.7: 2108242.0000 - fp0.7: 222942.0000 - tn0.7: 16460026.0000 - fn0.7: 869590.0000 - precision0
    .7: 0.9044 - recall0.7: 0.7080 - tp0.9: 1658131.0000 - fp0.9: 56354.0000 - tn0.9: 16626614.0000 - fn0.9: 1319701.0000 - precision0.9: 0.9671 - recall0.9: 0.5568 - accuracy: 0.9440 - auc: 0
    .9516 - f1: 0.2631 - val_loss: 1.4853 - val_tp0.1: 516528.0000 - val_fp0.1: 1043680.0000 - val_tn0.1: 3067663.0000 - val_fn0.1: 287329.0000 - val_precision0.1: 0.3311 - val_recall0.1: 0.64
    26 - val_tp0.3: 418011.0000 - val_fp0.3: 456854.0000 - val_tn0.3: 3654489.0000 - val_fn0.3: 385846.0000 - val_precision0.3: 0.4778 - val_recall0.3: 0.5200 - val_tp0.5: 350573.0000 - val_fp
    0.5: 253208.0000 - val_tn0.5: 3858135.0000 - val_fn0.5: 453284.0000 - val_precision0.5: 0.5806 - val_recall0.5: 0.4361 - val_tp0.7: 302626.0000 - val_fp0.7: 145263.0000 - val_tn0.7: 396608
    0.0000 - val_fn0.7: 501231.0000 - val_precision0.7: 0.6757 - val_recall0.7: 0.3765 - val_tp0.9: 245404.0000 - val_fp0.9: 47550.0000 - val_tn0.9: 4063793.0000 - val_fn0.9: 558453.0000 - val
    _precision0.9: 0.8377 - val_recall0.9: 0.3053 - val_accuracy: 0.8563 - val_auc: 0.7430 - val_f1: 0.2811
    Epoch 16/20
    2400/2400 [==============================] - 63s 26ms/step - loss: 0.3733 - tp0.1: 2769563.0000 - fp0.1: 1728895.0000 - tn0.1: 14954073.0000 - fn0.1: 208269.0000 - precision0.1: 0.6157 - r
    ecall0.1: 0.9301 - tp0.3: 2599312.0000 - fp0.3: 855827.0000 - tn0.3: 15827141.0000 - fn0.3: 378520.0000 - precision0.3: 0.7523 - recall0.3: 0.8729 - tp0.5: 2413813.0000 - fp0.5: 450780.000
    0 - tn0.5: 16232188.0000 - fn0.5: 564019.0000 - precision0.5: 0.8426 - recall0.5: 0.8106 - tp0.7: 2181800.0000 - fp0.7: 210158.0000 - tn0.7: 16472810.0000 - fn0.7: 796032.0000 - precision0
    .7: 0.9121 - recall0.7: 0.7327 - tp0.9: 1723463.0000 - fp0.9: 51642.0000 - tn0.9: 16631326.0000 - fn0.9: 1254369.0000 - precision0.9: 0.9709 - recall0.9: 0.5788 - accuracy: 0.9484 - auc: 0
    .9564 - f1: 0.2631 - val_loss: 9.3258 - val_tp0.1: 803646.0000 - val_fp0.1: 4110058.0000 - val_tn0.1: 1285.0000 - val_fn0.1: 211.0000 - val_precision0.1: 0.1636 - val_recall0.1: 0.9997 - v
    al_tp0.3: 803193.0000 - val_fp0.3: 4106394.0000 - val_tn0.3: 4949.0000 - val_fn0.3: 664.0000 - val_precision0.3: 0.1636 - val_recall0.3: 0.9992 - val_tp0.5: 802499.0000 - val_fp0.5: 410051
    0.0000 - val_tn0.5: 10833.0000 - val_fn0.5: 1358.0000 - val_precision0.5: 0.1637 - val_recall0.5: 0.9983 - val_tp0.7: 800907.0000 - val_fp0.7: 4088834.0000 - val_tn0.7: 22509.0000 - val_fn
    0.7: 2950.0000 - val_precision0.7: 0.1638 - val_recall0.7: 0.9963 - val_tp0.9: 794980.0000 - val_fp0.9: 4040625.0000 - val_tn0.9: 70718.0000 - val_fn0.9: 8877.0000 - val_precision0.9: 0.16
    44 - val_recall0.9: 0.9890 - val_accuracy: 0.1655 - val_auc: 0.5230 - val_f1: 0.2811
    Epoch 17/20
    2400/2400 [==============================] - 63s 26ms/step - loss: 0.3676 - tp0.1: 2769181.0000 - fp0.1: 1659194.0000 - tn0.1: 15023774.0000 - fn0.1: 208651.0000 - precision0.1: 0.6253 - r
    ecall0.1: 0.9299 - tp0.3: 2610951.0000 - fp0.3: 858713.0000 - tn0.3: 15824255.0000 - fn0.3: 366881.0000 - precision0.3: 0.7525 - recall0.3: 0.8768 - tp0.5: 2429189.0000 - fp0.5: 452947.000
    0 - tn0.5: 16230021.0000 - fn0.5: 548643.0000 - precision0.5: 0.8428 - recall0.5: 0.8158 - tp0.7: 2189579.0000 - fp0.7: 210955.0000 - tn0.7: 16472013.0000 - fn0.7: 788253.0000 - precision0
    .7: 0.9121 - recall0.7: 0.7353 - tp0.9: 1747880.0000 - fp0.9: 50128.0000 - tn0.9: 16632840.0000 - fn0.9: 1229952.0000 - precision0.9: 0.9721 - recall0.9: 0.5870 - accuracy: 0.9491 - auc: 0
    .9570 - f1: 0.2631 - val_loss: 5.5868 - val_tp0.1: 802997.0000 - val_fp0.1: 4076916.0000 - val_tn0.1: 34427.0000 - val_fn0.1: 860.0000 - val_precision0.1: 0.1646 - val_recall0.1: 0.9989 -
    val_tp0.3: 802425.0000 - val_fp0.3: 4060844.0000 - val_tn0.3: 50499.0000 - val_fn0.3: 1432.0000 - val_precision0.3: 0.1650 - val_recall0.3: 0.9982 - val_tp0.5: 801861.0000 - val_fp0.5: 404
    2504.0000 - val_tn0.5: 68839.0000 - val_fn0.5: 1996.0000 - val_precision0.5: 0.1655 - val_recall0.5: 0.9975 - val_tp0.7: 800923.0000 - val_fp0.7: 4011295.0000 - val_tn0.7: 100048.0000 - va
    l_fn0.7: 2934.0000 - val_precision0.7: 0.1664 - val_recall0.7: 0.9964 - val_tp0.9: 795379.0000 - val_fp0.9: 3913030.0000 - val_tn0.9: 198313.0000 - val_fn0.9: 8478.0000 - val_precision0.9:
     0.1689 - val_recall0.9: 0.9895 - val_accuracy: 0.1771 - val_auc: 0.5462 - val_f1: 0.2811
    Epoch 18/20
    2400/2400 [==============================] - 63s 26ms/step - loss: 0.3717 - tp0.1: 2768914.0000 - fp0.1: 1725165.0000 - tn0.1: 14957803.0000 - fn0.1: 208918.0000 - precision0.1: 0.6161 - r
    ecall0.1: 0.9298 - tp0.3: 2598193.0000 - fp0.3: 861638.0000 - tn0.3: 15821330.0000 - fn0.3: 379639.0000 - precision0.3: 0.7510 - recall0.3: 0.8725 - tp0.5: 2411640.0000 - fp0.5: 445576.000
    0 - tn0.5: 16237392.0000 - fn0.5: 566192.0000 - precision0.5: 0.8441 - recall0.5: 0.8099 - tp0.7: 2177668.0000 - fp0.7: 207154.0000 - tn0.7: 16475814.0000 - fn0.7: 800164.0000 - precision0
    .7: 0.9131 - recall0.7: 0.7313 - tp0.9: 1736637.0000 - fp0.9: 49527.0000 - tn0.9: 16633441.0000 - fn0.9: 1241195.0000 - precision0.9: 0.9723 - recall0.9: 0.5832 - accuracy: 0.9485 - auc: 0
    .9563 - f1: 0.2631 - val_loss: 1.5014 - val_tp0.1: 559528.0000 - val_fp0.1: 1842714.0000 - val_tn0.1: 2268629.0000 - val_fn0.1: 244329.0000 - val_precision0.1: 0.2329 - val_recall0.1: 0.69
    61 - val_tp0.3: 502697.0000 - val_fp0.3: 1389421.0000 - val_tn0.3: 2721922.0000 - val_fn0.3: 301160.0000 - val_precision0.3: 0.2657 - val_recall0.3: 0.6254 - val_tp0.5: 413701.0000 - val_f
    p0.5: 665286.0000 - val_tn0.5: 3446057.0000 - val_fn0.5: 390156.0000 - val_precision0.5: 0.3834 - val_recall0.5: 0.5146 - val_tp0.7: 345100.0000 - val_fp0.7: 322127.0000 - val_tn0.7: 37892
    16.0000 - val_fn0.7: 458757.0000 - val_precision0.7: 0.5172 - val_recall0.7: 0.4293 - val_tp0.9: 274540.0000 - val_fp0.9: 106793.0000 - val_tn0.9: 4004550.0000 - val_fn0.9: 529317.0000 - v
    al_precision0.9: 0.7199 - val_recall0.9: 0.3415 - val_accuracy: 0.7853 - val_auc: 0.7127 - val_f1: 0.2811
    Epoch 19/20
    2400/2400 [==============================] - 63s 26ms/step - loss: 0.3547 - tp0.1: 2782451.0000 - fp0.1: 1665578.0000 - tn0.1: 15017390.0000 - fn0.1: 195381.0000 - precision0.1: 0.6255 - r
    ecall0.1: 0.9344 - tp0.3: 2619177.0000 - fp0.3: 850031.0000 - tn0.3: 15832937.0000 - fn0.3: 358655.0000 - precision0.3: 0.7550 - recall0.3: 0.8796 - tp0.5: 2432227.0000 - fp0.5: 440386.000
    0 - tn0.5: 16242582.0000 - fn0.5: 545605.0000 - precision0.5: 0.8467 - recall0.5: 0.8168 - tp0.7: 2203211.0000 - fp0.7: 207630.0000 - tn0.7: 16475338.0000 - fn0.7: 774621.0000 - precision0
    .7: 0.9139 - recall0.7: 0.7399 - tp0.9: 1758972.0000 - fp0.9: 48353.0000 - tn0.9: 16634615.0000 - fn0.9: 1218860.0000 - precision0.9: 0.9732 - recall0.9: 0.5907 - accuracy: 0.9499 - auc: 0
    .9594 - f1: 0.2631 - val_loss: 1.6122 - val_tp0.1: 364867.0000 - val_fp0.1: 80402.0000 - val_tn0.1: 4030941.0000 - val_fn0.1: 438990.0000 - val_precision0.1: 0.8194 - val_recall0.1: 0.4539
     - val_tp0.3: 338563.0000 - val_fp0.3: 52446.0000 - val_tn0.3: 4058897.0000 - val_fn0.3: 465294.0000 - val_precision0.3: 0.8659 - val_recall0.3: 0.4212 - val_tp0.5: 303084.0000 - val_fp0.5
    : 26254.0000 - val_tn0.5: 4085089.0000 - val_fn0.5: 500773.0000 - val_precision0.5: 0.9203 - val_recall0.5: 0.3770 - val_tp0.7: 268991.0000 - val_fp0.7: 12371.0000 - val_tn0.7: 4098972.000
    0 - val_fn0.7: 534866.0000 - val_precision0.7: 0.9560 - val_recall0.7: 0.3346 - val_tp0.9: 228930.0000 - val_fp0.9: 4729.0000 - val_tn0.9: 4106614.0000 - val_fn0.9: 574927.0000 - val_preci
    sion0.9: 0.9798 - val_recall0.9: 0.2848 - val_accuracy: 0.8928 - val_auc: 0.7252 - val_f1: 0.2811
    Epoch 20/20
    2400/2400 [==============================] - 63s 26ms/step - loss: 0.3465 - tp0.1: 2789105.0000 - fp0.1: 1630391.0000 - tn0.1: 15052577.0000 - fn0.1: 188727.0000 - precision0.1: 0.6311 - r
    ecall0.1: 0.9366 - tp0.3: 2634590.0000 - fp0.3: 826424.0000 - tn0.3: 15856544.0000 - fn0.3: 343242.0000 - precision0.3: 0.7612 - recall0.3: 0.8847 - tp0.5: 2453024.0000 - fp0.5: 430471.000
    0 - tn0.5: 16252497.0000 - fn0.5: 524808.0000 - precision0.5: 0.8507 - recall0.5: 0.8238 - tp0.7: 2228295.0000 - fp0.7: 203336.0000 - tn0.7: 16479632.0000 - fn0.7: 749537.0000 - precision0
    .7: 0.9164 - recall0.7: 0.7483 - tp0.9: 1795290.0000 - fp0.9: 47888.0000 - tn0.9: 16635080.0000 - fn0.9: 1182542.0000 - precision0.9: 0.9740 - recall0.9: 0.6029 - accuracy: 0.9514 - auc: 0
    .9610 - f1: 0.2631 - val_loss: 1.6856 - val_tp0.1: 306188.0000 - val_fp0.1: 33472.0000 - val_tn0.1: 4077871.0000 - val_fn0.1: 497669.0000 - val_precision0.1: 0.9015 - val_recall0.1: 0.3809
     - val_tp0.3: 252310.0000 - val_fp0.3: 16537.0000 - val_tn0.3: 4094806.0000 - val_fn0.3: 551547.0000 - val_precision0.3: 0.9385 - val_recall0.3: 0.3139 - val_tp0.5: 216053.0000 - val_fp0.5
    : 9308.0000 - val_tn0.5: 4102035.0000 - val_fn0.5: 587804.0000 - val_precision0.5: 0.9587 - val_recall0.5: 0.2688 - val_tp0.7: 183648.0000 - val_fp0.7: 4739.0000 - val_tn0.7: 4106604.0000
    - val_fn0.7: 620209.0000 - val_precision0.7: 0.9748 - val_recall0.7: 0.2285 - val_tp0.9: 141556.0000 - val_fp0.9: 1421.0000 - val_tn0.9: 4109922.0000 - val_fn0.9: 662301.0000 - val_precisi
    on0.9: 0.9901 - val_recall0.9: 0.1761 - val_accuracy: 0.8785 - val_auc: 0.7019 - val_f1: 0.2811
    --- Running training session 55/140
    {'hp_epochs': 20, 'hp_batch_size': 10, 'hp_scaler': 'robust', 'hp_n_levels': 5, 'hp_first_filters': 16, 'hp_pool_size': 4, 'hp_input_size': 8192, 'hp_lr_start': 0.06276763366515732, 'hp_lr
    _power': 1.0}
    --- repeat #: 1
    input - shape:   (None, 8192, 1)
    output - shape:  (None, 8192, 1)
    Epoch 1/20
    480/480 [==============================] - 45s 53ms/step - loss: 0.7092 - tp0.1: 5321969.0000 - fp0.1: 6687422.0000 - tn0.1: 26697388.0000 - fn0.1: 614807.0000 - precision0.1: 0.4432 - rec
    all0.1: 0.8964 - tp0.3: 4646941.0000 - fp0.3: 2819314.0000 - tn0.3: 30565520.0000 - fn0.3: 1289835.0000 - precision0.3: 0.6224 - recall0.3: 0.7827 - tp0.5: 3895656.0000 - fp0.5: 1079912.00
    00 - tn0.5: 32304924.0000 - fn0.5: 2041120.0000 - precision0.5: 0.7830 - recall0.5: 0.6562 - tp0.7: 3225404.0000 - fp0.7: 417241.0000 - tn0.7: 32967584.0000 - fn0.7: 2711372.0000 - precisi
    on0.7: 0.8855 - recall0.7: 0.5433 - tp0.9: 2363272.0000 - fp0.9: 80854.0000 - tn0.9: 33303980.0000 - fn0.9: 3573504.0000 - precision0.9: 0.9669 - recall0.9: 0.3981 - accuracy: 0.9206 - auc
    : 0.9209 - f1: 0.2624 - val_loss: 1.7392 - val_tp0.1: 675173.0000 - val_fp0.1: 51103.0000 - val_tn0.1: 8196692.0000 - val_fn0.1: 907432.0000 - val_precision0.1: 0.9296 - val_recall0.1: 0.4
    266 - val_tp0.3: 629304.0000 - val_fp0.3: 27265.0000 - val_tn0.3: 8220530.0000 - val_fn0.3: 953301.0000 - val_precision0.3: 0.9585 - val_recall0.3: 0.3976 - val_tp0.5: 554968.0000 - val_fp
    0.5: 8081.0000 - val_tn0.5: 8239714.0000 - val_fn0.5: 1027637.0000 - val_precision0.5: 0.9856 - val_recall0.5: 0.3507 - val_tp0.7: 482396.0000 - val_fp0.7: 1352.0000 - val_tn0.7: 8246443.0
    000 - val_fn0.7: 1100209.0000 - val_precision0.7: 0.9972 - val_recall0.7: 0.3048 - val_tp0.9: 365703.0000 - val_fp0.9: 93.0000 - val_tn0.9: 8247702.0000 - val_fn0.9: 1216902.0000 - val_pre
    cision0.9: 0.9997 - val_recall0.9: 0.2311 - val_accuracy: 0.8946 - val_auc: 0.7140 - val_f1: 0.2773
    Epoch 2/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.5057 - tp0.1: 5497302.0000 - fp0.1: 6092525.0000 - tn0.1: 27292284.0000 - fn0.1: 439474.0000 - precision0.1: 0.4743 - rec
    all0.1: 0.9260 - tp0.3: 4832858.0000 - fp0.3: 2368876.0000 - tn0.3: 31015944.0000 - fn0.3: 1103918.0000 - precision0.3: 0.6711 - recall0.3: 0.8141 - tp0.5: 4194498.0000 - fp0.5: 1025948.00
    00 - tn0.5: 32358848.0000 - fn0.5: 1742278.0000 - precision0.5: 0.8035 - recall0.5: 0.7065 - tp0.7: 3542699.0000 - fp0.7: 455958.0000 - tn0.7: 32928884.0000 - fn0.7: 2394077.0000 - precisi
    on0.7: 0.8860 - recall0.7: 0.5967 - tp0.9: 2649175.0000 - fp0.9: 121325.0000 - tn0.9: 33263512.0000 - fn0.9: 3287601.0000 - precision0.9: 0.9562 - recall0.9: 0.4462 - accuracy: 0.9296 - au
    c: 0.9389 - f1: 0.2624 - val_loss: 1.7645 - val_tp0.1: 734127.0000 - val_fp0.1: 22185.0000 - val_tn0.1: 8225610.0000 - val_fn0.1: 848478.0000 - val_precision0.1: 0.9707 - val_recall0.1: 0.
    4639 - val_tp0.3: 692477.0000 - val_fp0.3: 8041.0000 - val_tn0.3: 8239754.0000 - val_fn0.3: 890128.0000 - val_precision0.3: 0.9885 - val_recall0.3: 0.4376 - val_tp0.5: 645446.0000 - val_fp
    0.5: 3920.0000 - val_tn0.5: 8243875.0000 - val_fn0.5: 937159.0000 - val_precision0.5: 0.9940 - val_recall0.5: 0.4078 - val_tp0.7: 574282.0000 - val_fp0.7: 1352.0000 - val_tn0.7: 8246443.00
    00 - val_fn0.7: 1008323.0000 - val_precision0.7: 0.9977 - val_recall0.7: 0.3629 - val_tp0.9: 453034.0000 - val_fp0.9: 218.0000 - val_tn0.9: 8247577.0000 - val_fn0.9: 1129571.0000 - val_pre
    cision0.9: 0.9995 - val_recall0.9: 0.2863 - val_accuracy: 0.9043 - val_auc: 0.7331 - val_f1: 0.2773
    Epoch 3/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.4505 - tp0.1: 5568258.0000 - fp0.1: 5463889.0000 - tn0.1: 27920928.0000 - fn0.1: 368518.0000 - precision0.1: 0.5047 - rec
    all0.1: 0.9379 - tp0.3: 4929910.0000 - fp0.3: 2020777.0000 - tn0.3: 31364036.0000 - fn0.3: 1006866.0000 - precision0.3: 0.7093 - recall0.3: 0.8304 - tp0.5: 4449095.0000 - fp0.5: 1041240.00
    00 - tn0.5: 32343572.0000 - fn0.5: 1487681.0000 - precision0.5: 0.8104 - recall0.5: 0.7494 - tp0.7: 3823844.0000 - fp0.7: 479840.0000 - tn0.7: 32904986.0000 - fn0.7: 2112932.0000 - precisi
    on0.7: 0.8885 - recall0.7: 0.6441 - tp0.9: 2882785.0000 - fp0.9: 126351.0000 - tn0.9: 33258496.0000 - fn0.9: 3053991.0000 - precision0.9: 0.9580 - recall0.9: 0.4856 - accuracy: 0.9357 - au
    c: 0.9475 - f1: 0.2624 - val_loss: 0.6952 - val_tp0.1: 1307421.0000 - val_fp0.1: 509799.0000 - val_tn0.1: 7737996.0000 - val_fn0.1: 275184.0000 - val_precision0.1: 0.7195 - val_recall0.1:
    0.8261 - val_tp0.3: 1160591.0000 - val_fp0.3: 197075.0000 - val_tn0.3: 8050720.0000 - val_fn0.3: 422014.0000 - val_precision0.3: 0.8548 - val_recall0.3: 0.7333 - val_tp0.5: 946439.0000 - v
    al_fp0.5: 19008.0000 - val_tn0.5: 8228787.0000 - val_fn0.5: 636166.0000 - val_precision0.5: 0.9803 - val_recall0.5: 0.5980 - val_tp0.7: 695971.0000 - val_fp0.7: 2696.0000 - val_tn0.7: 8245
    099.0000 - val_fn0.7: 886634.0000 - val_precision0.7: 0.9961 - val_recall0.7: 0.4398 - val_tp0.9: 409103.0000 - val_fp0.9: 135.0000 - val_tn0.9: 8247660.0000 - val_fn0.9: 1173502.0000 - va
    l_precision0.9: 0.9997 - val_recall0.9: 0.2585 - val_accuracy: 0.9334 - val_auc: 0.9043 - val_f1: 0.2773
    Epoch 4/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.4147 - tp0.1: 5582837.0000 - fp0.1: 4800957.0000 - tn0.1: 28583876.0000 - fn0.1: 353939.0000 - precision0.1: 0.5376 - rec
    all0.1: 0.9404 - tp0.3: 5025077.0000 - fp0.3: 1931724.0000 - tn0.3: 31453094.0000 - fn0.3: 911699.0000 - precision0.3: 0.7223 - recall0.3: 0.8464 - tp0.5: 4590126.0000 - fp0.5: 953938.0000
     - tn0.5: 32430896.0000 - fn0.5: 1346650.0000 - precision0.5: 0.8279 - recall0.5: 0.7732 - tp0.7: 4038584.0000 - fp0.7: 465004.0000 - tn0.7: 32919816.0000 - fn0.7: 1898192.0000 - precision
    0.7: 0.8967 - recall0.7: 0.6803 - tp0.9: 3025476.0000 - fp0.9: 119273.0000 - tn0.9: 33265544.0000 - fn0.9: 2911300.0000 - precision0.9: 0.9621 - recall0.9: 0.5096 - accuracy: 0.9415 - auc:
     0.9527 - f1: 0.2624 - val_loss: 0.5201 - val_tp0.1: 1399126.0000 - val_fp0.1: 622073.0000 - val_tn0.1: 7625722.0000 - val_fn0.1: 183479.0000 - val_precision0.1: 0.6922 - val_recall0.1: 0.
    8841 - val_tp0.3: 1113548.0000 - val_fp0.3: 36502.0000 - val_tn0.3: 8211293.0000 - val_fn0.3: 469057.0000 - val_precision0.3: 0.9683 - val_recall0.3: 0.7036 - val_tp0.5: 884202.0000 - val_
    fp0.5: 4823.0000 - val_tn0.5: 8242972.0000 - val_fn0.5: 698403.0000 - val_precision0.5: 0.9946 - val_recall0.5: 0.5587 - val_tp0.7: 569297.0000 - val_fp0.7: 628.0000 - val_tn0.7: 8247167.0
    000 - val_fn0.7: 1013308.0000 - val_precision0.7: 0.9989 - val_recall0.7: 0.3597 - val_tp0.9: 374108.0000 - val_fp0.9: 50.0000 - val_tn0.9: 8247745.0000 - val_fn0.9: 1208497.0000 - val_pre
    cision0.9: 0.9999 - val_recall0.9: 0.2364 - val_accuracy: 0.9285 - val_auc: 0.9333 - val_f1: 0.2773
    Epoch 5/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.3754 - tp0.1: 5634883.0000 - fp0.1: 4427128.0000 - tn0.1: 28957694.0000 - fn0.1: 301893.0000 - precision0.1: 0.5600 - rec
    all0.1: 0.9491 - tp0.3: 5093958.0000 - fp0.3: 1784747.0000 - tn0.3: 31600080.0000 - fn0.3: 842818.0000 - precision0.3: 0.7405 - recall0.3: 0.8580 - tp0.5: 4689585.0000 - fp0.5: 889979.0000
     - tn0.5: 32494848.0000 - fn0.5: 1247191.0000 - precision0.5: 0.8405 - recall0.5: 0.7899 - tp0.7: 4192223.0000 - fp0.7: 451284.0000 - tn0.7: 32933542.0000 - fn0.7: 1744553.0000 - precision
    0.7: 0.9028 - recall0.7: 0.7061 - tp0.9: 3191877.0000 - fp0.9: 120760.0000 - tn0.9: 33264048.0000 - fn0.9: 2744899.0000 - precision0.9: 0.9635 - recall0.9: 0.5376 - accuracy: 0.9456 - auc:
     0.9584 - f1: 0.2624 - val_loss: 0.4916 - val_tp0.1: 1492248.0000 - val_fp0.1: 1196998.0000 - val_tn0.1: 7050797.0000 - val_fn0.1: 90357.0000 - val_precision0.1: 0.5549 - val_recall0.1: 0.
    9429 - val_tp0.3: 948440.0000 - val_fp0.3: 76049.0000 - val_tn0.3: 8171746.0000 - val_fn0.3: 634165.0000 - val_precision0.3: 0.9258 - val_recall0.3: 0.5993 - val_tp0.5: 790703.0000 - val_f
    p0.5: 18060.0000 - val_tn0.5: 8229735.0000 - val_fn0.5: 791902.0000 - val_precision0.5: 0.9777 - val_recall0.5: 0.4996 - val_tp0.7: 644735.0000 - val_fp0.7: 3337.0000 - val_tn0.7: 8244458.
    0000 - val_fn0.7: 937870.0000 - val_precision0.7: 0.9949 - val_recall0.7: 0.4074 - val_tp0.9: 446636.0000 - val_fp0.9: 194.0000 - val_tn0.9: 8247601.0000 - val_fn0.9: 1135969.0000 - val_pr
    ecision0.9: 0.9996 - val_recall0.9: 0.2822 - val_accuracy: 0.9176 - val_auc: 0.9509 - val_f1: 0.2773
    Epoch 6/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.3576 - tp0.1: 5650782.0000 - fp0.1: 4214930.0000 - tn0.1: 29169904.0000 - fn0.1: 285994.0000 - precision0.1: 0.5728 - rec
    all0.1: 0.9518 - tp0.3: 5157731.0000 - fp0.3: 1802746.0000 - tn0.3: 31582076.0000 - fn0.3: 779045.0000 - precision0.3: 0.7410 - recall0.3: 0.8688 - tp0.5: 4706560.0000 - fp0.5: 815239.0000
     - tn0.5: 32569602.0000 - fn0.5: 1230216.0000 - precision0.5: 0.8524 - recall0.5: 0.7928 - tp0.7: 4257269.0000 - fp0.7: 416618.0000 - tn0.7: 32968196.0000 - fn0.7: 1679507.0000 - precision
    0.7: 0.9109 - recall0.7: 0.7171 - tp0.9: 3338555.0000 - fp0.9: 117993.0000 - tn0.9: 33266826.0000 - fn0.9: 2598221.0000 - precision0.9: 0.9659 - recall0.9: 0.5624 - accuracy: 0.9480 - auc:
     0.9615 - f1: 0.2624 - val_loss: 0.5022 - val_tp0.1: 1476921.0000 - val_fp0.1: 1294149.0000 - val_tn0.1: 6953646.0000 - val_fn0.1: 105684.0000 - val_precision0.1: 0.5330 - val_recall0.1: 0
    .9332 - val_tp0.3: 1409625.0000 - val_fp0.3: 897663.0000 - val_tn0.3: 7350132.0000 - val_fn0.3: 172980.0000 - val_precision0.3: 0.6109 - val_recall0.3: 0.8907 - val_tp0.5: 1288431.0000 - v
    al_fp0.5: 516719.0000 - val_tn0.5: 7731076.0000 - val_fn0.5: 294174.0000 - val_precision0.5: 0.7138 - val_recall0.5: 0.8141 - val_tp0.7: 1184168.0000 - val_fp0.7: 362179.0000 - val_tn0.7:
    7885616.0000 - val_fn0.7: 398437.0000 - val_precision0.7: 0.7658 - val_recall0.7: 0.7482 - val_tp0.9: 982390.0000 - val_fp0.9: 193676.0000 - val_tn0.9: 8054119.0000 - val_fn0.9: 600215.000
    0 - val_precision0.9: 0.8353 - val_recall0.9: 0.6207 - val_accuracy: 0.9175 - val_auc: 0.9394 - val_f1: 0.2773
    Epoch 7/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.3384 - tp0.1: 5645527.0000 - fp0.1: 3726823.0000 - tn0.1: 29657992.0000 - fn0.1: 291249.0000 - precision0.1: 0.6024 - rec
    all0.1: 0.9509 - tp0.3: 5236539.0000 - fp0.3: 1794315.0000 - tn0.3: 31590506.0000 - fn0.3: 700237.0000 - precision0.3: 0.7448 - recall0.3: 0.8821 - tp0.5: 4787739.0000 - fp0.5: 779171.0000
     - tn0.5: 32605672.0000 - fn0.5: 1149037.0000 - precision0.5: 0.8600 - recall0.5: 0.8065 - tp0.7: 4390618.0000 - fp0.7: 404832.0000 - tn0.7: 32980006.0000 - fn0.7: 1546158.0000 - precision
    0.7: 0.9156 - recall0.7: 0.7396 - tp0.9: 3468567.0000 - fp0.9: 102570.0000 - tn0.9: 33282264.0000 - fn0.9: 2468209.0000 - precision0.9: 0.9713 - recall0.9: 0.5843 - accuracy: 0.9510 - auc:
     0.9638 - f1: 0.2624 - val_loss: 0.3821 - val_tp0.1: 1458280.0000 - val_fp0.1: 507037.0000 - val_tn0.1: 7740758.0000 - val_fn0.1: 124325.0000 - val_precision0.1: 0.7420 - val_recall0.1: 0.
    9214 - val_tp0.3: 1138703.0000 - val_fp0.3: 109160.0000 - val_tn0.3: 8138635.0000 - val_fn0.3: 443902.0000 - val_precision0.3: 0.9125 - val_recall0.3: 0.7195 - val_tp0.5: 963399.0000 - val
    _fp0.5: 44369.0000 - val_tn0.5: 8203426.0000 - val_fn0.5: 619206.0000 - val_precision0.5: 0.9560 - val_recall0.5: 0.6087 - val_tp0.7: 768098.0000 - val_fp0.7: 13185.0000 - val_tn0.7: 82346
    10.0000 - val_fn0.7: 814507.0000 - val_precision0.7: 0.9831 - val_recall0.7: 0.4853 - val_tp0.9: 549365.0000 - val_fp0.9: 1330.0000 - val_tn0.9: 8246465.0000 - val_fn0.9: 1033240.0000 - va
    l_precision0.9: 0.9976 - val_recall0.9: 0.3471 - val_accuracy: 0.9325 - val_auc: 0.9604 - val_f1: 0.2773
    Epoch 8/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.3065 - tp0.1: 5676575.0000 - fp0.1: 3449099.0000 - tn0.1: 29935738.0000 - fn0.1: 260201.0000 - precision0.1: 0.6220 - rec
    all0.1: 0.9562 - tp0.3: 5325689.0000 - fp0.3: 1652021.0000 - tn0.3: 31732808.0000 - fn0.3: 611087.0000 - precision0.3: 0.7632 - recall0.3: 0.8971 - tp0.5: 4859260.0000 - fp0.5: 655135.0000
     - tn0.5: 32729690.0000 - fn0.5: 1077516.0000 - precision0.5: 0.8812 - recall0.5: 0.8185 - tp0.7: 4528001.0000 - fp0.7: 355303.0000 - tn0.7: 33029518.0000 - fn0.7: 1408775.0000 - precision
    0.7: 0.9272 - recall0.7: 0.7627 - tp0.9: 3797501.0000 - fp0.9: 104696.0000 - tn0.9: 33280128.0000 - fn0.9: 2139275.0000 - precision0.9: 0.9732 - recall0.9: 0.6397 - accuracy: 0.9559 - auc:
     0.9683 - f1: 0.2624 - val_loss: 0.4780 - val_tp0.1: 1324114.0000 - val_fp0.1: 275210.0000 - val_tn0.1: 7972585.0000 - val_fn0.1: 258491.0000 - val_precision0.1: 0.8279 - val_recall0.1: 0.
    8367 - val_tp0.3: 1141828.0000 - val_fp0.3: 54316.0000 - val_tn0.3: 8193479.0000 - val_fn0.3: 440777.0000 - val_precision0.3: 0.9546 - val_recall0.3: 0.7215 - val_tp0.5: 1048254.0000 - val
    _fp0.5: 27301.0000 - val_tn0.5: 8220494.0000 - val_fn0.5: 534351.0000 - val_precision0.5: 0.9746 - val_recall0.5: 0.6624 - val_tp0.7: 938136.0000 - val_fp0.7: 13368.0000 - val_tn0.7: 82344
    27.0000 - val_fn0.7: 644469.0000 - val_precision0.7: 0.9860 - val_recall0.7: 0.5928 - val_tp0.9: 734879.0000 - val_fp0.9: 3835.0000 - val_tn0.9: 8243960.0000 - val_fn0.9: 847726.0000 - val
    _precision0.9: 0.9948 - val_recall0.9: 0.4643 - val_accuracy: 0.9429 - val_auc: 0.9295 - val_f1: 0.2773
    Epoch 9/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.3010 - tp0.1: 5676976.0000 - fp0.1: 3336526.0000 - tn0.1: 30048290.0000 - fn0.1: 259800.0000 - precision0.1: 0.6298 - rec
    all0.1: 0.9562 - tp0.3: 5326144.0000 - fp0.3: 1579793.0000 - tn0.3: 31805038.0000 - fn0.3: 610632.0000 - precision0.3: 0.7712 - recall0.3: 0.8971 - tp0.5: 4898244.0000 - fp0.5: 689875.0000
     - tn0.5: 32694952.0000 - fn0.5: 1038532.0000 - precision0.5: 0.8765 - recall0.5: 0.8251 - tp0.7: 4562202.0000 - fp0.7: 365675.0000 - tn0.7: 33019140.0000 - fn0.7: 1374574.0000 - precision
    0.7: 0.9258 - recall0.7: 0.7685 - tp0.9: 3784831.0000 - fp0.9: 104828.0000 - tn0.9: 33280008.0000 - fn0.9: 2151945.0000 - precision0.9: 0.9730 - recall0.9: 0.6375 - accuracy: 0.9560 - auc:
     0.9694 - f1: 0.2624 - val_loss: 0.4138 - val_tp0.1: 1542379.0000 - val_fp0.1: 1309329.0000 - val_tn0.1: 6938466.0000 - val_fn0.1: 40226.0000 - val_precision0.1: 0.5409 - val_recall0.1: 0.
    9746 - val_tp0.3: 1522585.0000 - val_fp0.3: 940842.0000 - val_tn0.3: 7306953.0000 - val_fn0.3: 60020.0000 - val_precision0.3: 0.6181 - val_recall0.3: 0.9621 - val_tp0.5: 1471806.0000 - val
    _fp0.5: 673062.0000 - val_tn0.5: 7574733.0000 - val_fn0.5: 110799.0000 - val_precision0.5: 0.6862 - val_recall0.5: 0.9300 - val_tp0.7: 1439073.0000 - val_fp0.7: 504700.0000 - val_tn0.7: 77
    43095.0000 - val_fn0.7: 143532.0000 - val_precision0.7: 0.7404 - val_recall0.7: 0.9093 - val_tp0.9: 1349827.0000 - val_fp0.9: 275289.0000 - val_tn0.9: 7972506.0000 - val_fn0.9: 232778.0000
     - val_precision0.9: 0.8306 - val_recall0.9: 0.8529 - val_accuracy: 0.9203 - val_auc: 0.9720 - val_f1: 0.2773
    Epoch 10/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2783 - tp0.1: 5687717.0000 - fp0.1: 2979751.0000 - tn0.1: 30405060.0000 - fn0.1: 249059.0000 - precision0.1: 0.6562 - rec
    all0.1: 0.9580 - tp0.3: 5401647.0000 - fp0.3: 1520293.0000 - tn0.3: 31864510.0000 - fn0.3: 535129.0000 - precision0.3: 0.7804 - recall0.3: 0.9099 - tp0.5: 4984841.0000 - fp0.5: 648600.0000
     - tn0.5: 32736208.0000 - fn0.5: 951935.0000 - precision0.5: 0.8849 - recall0.5: 0.8397 - tp0.7: 4668237.0000 - fp0.7: 338935.0000 - tn0.7: 33045886.0000 - fn0.7: 1268539.0000 - precision0
    .7: 0.9323 - recall0.7: 0.7863 - tp0.9: 3974497.0000 - fp0.9: 95618.0000 - tn0.9: 33289212.0000 - fn0.9: 1962279.0000 - precision0.9: 0.9765 - recall0.9: 0.6695 - accuracy: 0.9593 - auc: 0
    .9717 - f1: 0.2624 - val_loss: 0.3033 - val_tp0.1: 1528072.0000 - val_fp0.1: 891324.0000 - val_tn0.1: 7356471.0000 - val_fn0.1: 54533.0000 - val_precision0.1: 0.6316 - val_recall0.1: 0.965
    5 - val_tp0.3: 1410892.0000 - val_fp0.3: 321988.0000 - val_tn0.3: 7925807.0000 - val_fn0.3: 171713.0000 - val_precision0.3: 0.8142 - val_recall0.3: 0.8915 - val_tp0.5: 1288259.0000 - val_f
    p0.5: 124605.0000 - val_tn0.5: 8123190.0000 - val_fn0.5: 294346.0000 - val_precision0.5: 0.9118 - val_recall0.5: 0.8140 - val_tp0.7: 1162405.0000 - val_fp0.7: 53667.0000 - val_tn0.7: 81941
    28.0000 - val_fn0.7: 420200.0000 - val_precision0.7: 0.9559 - val_recall0.7: 0.7345 - val_tp0.9: 912340.0000 - val_fp0.9: 11304.0000 - val_tn0.9: 8236491.0000 - val_fn0.9: 670265.0000 - va
    l_precision0.9: 0.9878 - val_recall0.9: 0.5765 - val_accuracy: 0.9574 - val_auc: 0.9736 - val_f1: 0.2773
    Epoch 11/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2692 - tp0.1: 5698307.0000 - fp0.1: 2926299.0000 - tn0.1: 30458524.0000 - fn0.1: 238469.0000 - precision0.1: 0.6607 - rec
    all0.1: 0.9598 - tp0.3: 5414811.0000 - fp0.3: 1480385.0000 - tn0.3: 31904448.0000 - fn0.3: 521965.0000 - precision0.3: 0.7853 - recall0.3: 0.9121 - tp0.5: 4973314.0000 - fp0.5: 587269.0000
     - tn0.5: 32797544.0000 - fn0.5: 963462.0000 - precision0.5: 0.8944 - recall0.5: 0.8377 - tp0.7: 4687698.0000 - fp0.7: 318968.0000 - tn0.7: 33065868.0000 - fn0.7: 1249078.0000 - precision0
    .7: 0.9363 - recall0.7: 0.7896 - tp0.9: 4069138.0000 - fp0.9: 90368.0000 - tn0.9: 33294464.0000 - fn0.9: 1867638.0000 - precision0.9: 0.9783 - recall0.9: 0.6854 - accuracy: 0.9606 - auc: 0
    .9727 - f1: 0.2624 - val_loss: 0.2832 - val_tp0.1: 1495971.0000 - val_fp0.1: 516145.0000 - val_tn0.1: 7731650.0000 - val_fn0.1: 86634.0000 - val_precision0.1: 0.7435 - val_recall0.1: 0.945
    3 - val_tp0.3: 1379197.0000 - val_fp0.3: 185244.0000 - val_tn0.3: 8062551.0000 - val_fn0.3: 203408.0000 - val_precision0.3: 0.8816 - val_recall0.3: 0.8715 - val_tp0.5: 1284245.0000 - val_f
    p0.5: 85898.0000 - val_tn0.5: 8161897.0000 - val_fn0.5: 298360.0000 - val_precision0.5: 0.9373 - val_recall0.5: 0.8115 - val_tp0.7: 1199963.0000 - val_fp0.7: 43094.0000 - val_tn0.7: 820470
    1.0000 - val_fn0.7: 382642.0000 - val_precision0.7: 0.9653 - val_recall0.7: 0.7582 - val_tp0.9: 968381.0000 - val_fp0.9: 8568.0000 - val_tn0.9: 8239227.0000 - val_fn0.9: 614224.0000 - val_
    precision0.9: 0.9912 - val_recall0.9: 0.6119 - val_accuracy: 0.9609 - val_auc: 0.9676 - val_f1: 0.2773
    Epoch 12/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2531 - tp0.1: 5703005.0000 - fp0.1: 2614169.0000 - tn0.1: 30770672.0000 - fn0.1: 233771.0000 - precision0.1: 0.6857 - rec
    all0.1: 0.9606 - tp0.3: 5460500.0000 - fp0.3: 1362191.0000 - tn0.3: 32022638.0000 - fn0.3: 476276.0000 - precision0.3: 0.8003 - recall0.3: 0.9198 - tp0.5: 5066536.0000 - fp0.5: 594270.0000
     - tn0.5: 32790576.0000 - fn0.5: 870240.0000 - precision0.5: 0.8950 - recall0.5: 0.8534 - tp0.7: 4782731.0000 - fp0.7: 324884.0000 - tn0.7: 33059918.0000 - fn0.7: 1154045.0000 - precision0
    .7: 0.9364 - recall0.7: 0.8056 - tp0.9: 4134257.0000 - fp0.9: 91592.0000 - tn0.9: 33293228.0000 - fn0.9: 1802519.0000 - precision0.9: 0.9783 - recall0.9: 0.6964 - accuracy: 0.9628 - auc: 0
    .9742 - f1: 0.2624 - val_loss: 0.2880 - val_tp0.1: 1541566.0000 - val_fp0.1: 930091.0000 - val_tn0.1: 7317704.0000 - val_fn0.1: 41039.0000 - val_precision0.1: 0.6237 - val_recall0.1: 0.974
    1 - val_tp0.3: 1476199.0000 - val_fp0.3: 465005.0000 - val_tn0.3: 7782790.0000 - val_fn0.3: 106406.0000 - val_precision0.3: 0.7605 - val_recall0.3: 0.9328 - val_tp0.5: 1399471.0000 - val_f
    p0.5: 255679.0000 - val_tn0.5: 7992116.0000 - val_fn0.5: 183134.0000 - val_precision0.5: 0.8455 - val_recall0.5: 0.8843 - val_tp0.7: 1343272.0000 - val_fp0.7: 156828.0000 - val_tn0.7: 8090
    967.0000 - val_fn0.7: 239333.0000 - val_precision0.7: 0.8955 - val_recall0.7: 0.8488 - val_tp0.9: 1199567.0000 - val_fp0.9: 59158.0000 - val_tn0.9: 8188637.0000 - val_fn0.9: 383038.0000 -
    val_precision0.9: 0.9530 - val_recall0.9: 0.7580 - val_accuracy: 0.9554 - val_auc: 0.9778 - val_f1: 0.2773
    Epoch 13/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2610 - tp0.1: 5692281.0000 - fp0.1: 2715173.0000 - tn0.1: 30669648.0000 - fn0.1: 244495.0000 - precision0.1: 0.6771 - rec
    all0.1: 0.9588 - tp0.3: 5444152.0000 - fp0.3: 1405545.0000 - tn0.3: 31979284.0000 - fn0.3: 492624.0000 - precision0.3: 0.7948 - recall0.3: 0.9170 - tp0.5: 5067232.0000 - fp0.5: 645126.0000
     - tn0.5: 32739708.0000 - fn0.5: 869544.0000 - precision0.5: 0.8871 - recall0.5: 0.8535 - tp0.7: 4741815.0000 - fp0.7: 326101.0000 - tn0.7: 33058726.0000 - fn0.7: 1194961.0000 - precision0
    .7: 0.9357 - recall0.7: 0.7987 - tp0.9: 4095017.0000 - fp0.9: 90536.0000 - tn0.9: 33294300.0000 - fn0.9: 1841759.0000 - precision0.9: 0.9784 - recall0.9: 0.6898 - accuracy: 0.9615 - auc: 0
    .9733 - f1: 0.2624 - val_loss: 0.3264 - val_tp0.1: 1552497.0000 - val_fp0.1: 1332139.0000 - val_tn0.1: 6915656.0000 - val_fn0.1: 30108.0000 - val_precision0.1: 0.5382 - val_recall0.1: 0.98
    10 - val_tp0.3: 1491410.0000 - val_fp0.3: 604265.0000 - val_tn0.3: 7643530.0000 - val_fn0.3: 91195.0000 - val_precision0.3: 0.7117 - val_recall0.3: 0.9424 - val_tp0.5: 1423905.0000 - val_f
    p0.5: 336821.0000 - val_tn0.5: 7910974.0000 - val_fn0.5: 158700.0000 - val_precision0.5: 0.8087 - val_recall0.5: 0.8997 - val_tp0.7: 1364436.0000 - val_fp0.7: 196850.0000 - val_tn0.7: 8050
    945.0000 - val_fn0.7: 218169.0000 - val_precision0.7: 0.8739 - val_recall0.7: 0.8621 - val_tp0.9: 1214453.0000 - val_fp0.9: 61872.0000 - val_tn0.9: 8185923.0000 - val_fn0.9: 368152.0000 -
    val_precision0.9: 0.9515 - val_recall0.9: 0.7674 - val_accuracy: 0.9496 - val_auc: 0.9794 - val_f1: 0.2773
    Epoch 14/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2416 - tp0.1: 5720003.0000 - fp0.1: 2566509.0000 - tn0.1: 30818316.0000 - fn0.1: 216773.0000 - precision0.1: 0.6903 - rec
    all0.1: 0.9635 - tp0.3: 5480611.0000 - fp0.3: 1304768.0000 - tn0.3: 32080068.0000 - fn0.3: 456165.0000 - precision0.3: 0.8077 - recall0.3: 0.9232 - tp0.5: 5185408.0000 - fp0.5: 690719.0000
     - tn0.5: 32694086.0000 - fn0.5: 751368.0000 - precision0.5: 0.8825 - recall0.5: 0.8734 - tp0.7: 4822441.0000 - fp0.7: 319155.0000 - tn0.7: 33065634.0000 - fn0.7: 1114335.0000 - precision0
    .7: 0.9379 - recall0.7: 0.8123 - tp0.9: 4172445.0000 - fp0.9: 86620.0000 - tn0.9: 33298202.0000 - fn0.9: 1764331.0000 - precision0.9: 0.9797 - recall0.9: 0.7028 - accuracy: 0.9633 - auc: 0
    .9762 - f1: 0.2624 - val_loss: 0.3018 - val_tp0.1: 1466734.0000 - val_fp0.1: 405231.0000 - val_tn0.1: 7842564.0000 - val_fn0.1: 115871.0000 - val_precision0.1: 0.7835 - val_recall0.1: 0.92
    68 - val_tp0.3: 1381330.0000 - val_fp0.3: 201920.0000 - val_tn0.3: 8045875.0000 - val_fn0.3: 201275.0000 - val_precision0.3: 0.8725 - val_recall0.3: 0.8728 - val_tp0.5: 1281231.0000 - val_
    fp0.5: 78305.0000 - val_tn0.5: 8169490.0000 - val_fn0.5: 301374.0000 - val_precision0.5: 0.9424 - val_recall0.5: 0.8096 - val_tp0.7: 1204999.0000 - val_fp0.7: 38474.0000 - val_tn0.7: 82093
    21.0000 - val_fn0.7: 377606.0000 - val_precision0.7: 0.9691 - val_recall0.7: 0.7614 - val_tp0.9: 1069398.0000 - val_fp0.9: 12845.0000 - val_tn0.9: 8234950.0000 - val_fn0.9: 513207.0000 - v
    al_precision0.9: 0.9881 - val_recall0.9: 0.6757 - val_accuracy: 0.9614 - val_auc: 0.9591 - val_f1: 0.2773
    Epoch 15/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2295 - tp0.1: 5722432.0000 - fp0.1: 2370696.0000 - tn0.1: 31014116.0000 - fn0.1: 214344.0000 - precision0.1: 0.7071 - rec
    all0.1: 0.9639 - tp0.3: 5503430.0000 - fp0.3: 1205741.0000 - tn0.3: 32179086.0000 - fn0.3: 433346.0000 - precision0.3: 0.8203 - recall0.3: 0.9270 - tp0.5: 5226789.0000 - fp0.5: 622292.0000
     - tn0.5: 32762524.0000 - fn0.5: 709987.0000 - precision0.5: 0.8936 - recall0.5: 0.8804 - tp0.7: 4882199.0000 - fp0.7: 299113.0000 - tn0.7: 33085700.0000 - fn0.7: 1054577.0000 - precision0
    .7: 0.9423 - recall0.7: 0.8224 - tp0.9: 4261326.0000 - fp0.9: 78340.0000 - tn0.9: 33306484.0000 - fn0.9: 1675450.0000 - precision0.9: 0.9819 - recall0.9: 0.7178 - accuracy: 0.9661 - auc: 0
    .9771 - f1: 0.2624 - val_loss: 0.2662 - val_tp0.1: 1543285.0000 - val_fp0.1: 895817.0000 - val_tn0.1: 7351978.0000 - val_fn0.1: 39320.0000 - val_precision0.1: 0.6327 - val_recall0.1: 0.975
    2 - val_tp0.3: 1493466.0000 - val_fp0.3: 427226.0000 - val_tn0.3: 7820569.0000 - val_fn0.3: 89139.0000 - val_precision0.3: 0.7776 - val_recall0.3: 0.9437 - val_tp0.5: 1429946.0000 - val_fp
    0.5: 249890.0000 - val_tn0.5: 7997905.0000 - val_fn0.5: 152659.0000 - val_precision0.5: 0.8512 - val_recall0.5: 0.9035 - val_tp0.7: 1335056.0000 - val_fp0.7: 132081.0000 - val_tn0.7: 81157
    14.0000 - val_fn0.7: 247549.0000 - val_precision0.7: 0.9100 - val_recall0.7: 0.8436 - val_tp0.9: 1178764.0000 - val_fp0.9: 50269.0000 - val_tn0.9: 8197526.0000 - val_fn0.9: 403841.0000 - v
    al_precision0.9: 0.9591 - val_recall0.9: 0.7448 - val_accuracy: 0.9591 - val_auc: 0.9796 - val_f1: 0.2773
    Epoch 16/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2200 - tp0.1: 5727744.0000 - fp0.1: 2233811.0000 - tn0.1: 31151016.0000 - fn0.1: 209032.0000 - precision0.1: 0.7194 - rec
    all0.1: 0.9648 - tp0.3: 5535079.0000 - fp0.3: 1172465.0000 - tn0.3: 32212368.0000 - fn0.3: 401697.0000 - precision0.3: 0.8252 - recall0.3: 0.9323 - tp0.5: 5279991.0000 - fp0.5: 630115.0000
     - tn0.5: 32754706.0000 - fn0.5: 656785.0000 - precision0.5: 0.8934 - recall0.5: 0.8894 - tp0.7: 4922795.0000 - fp0.7: 289713.0000 - tn0.7: 33095104.0000 - fn0.7: 1013981.0000 - precision0
    .7: 0.9444 - recall0.7: 0.8292 - tp0.9: 4339994.0000 - fp0.9: 81950.0000 - tn0.9: 33302856.0000 - fn0.9: 1596782.0000 - precision0.9: 0.9815 - recall0.9: 0.7310 - accuracy: 0.9673 - auc: 0
    .9780 - f1: 0.2624 - val_loss: 0.3052 - val_tp0.1: 1502027.0000 - val_fp0.1: 713668.0000 - val_tn0.1: 7534127.0000 - val_fn0.1: 80578.0000 - val_precision0.1: 0.6779 - val_recall0.1: 0.949
    1 - val_tp0.3: 1373186.0000 - val_fp0.3: 249430.0000 - val_tn0.3: 7998365.0000 - val_fn0.3: 209419.0000 - val_precision0.3: 0.8463 - val_recall0.3: 0.8677 - val_tp0.5: 1287724.0000 - val_f
    p0.5: 120697.0000 - val_tn0.5: 8127098.0000 - val_fn0.5: 294881.0000 - val_precision0.5: 0.9143 - val_recall0.5: 0.8137 - val_tp0.7: 1203991.0000 - val_fp0.7: 63408.0000 - val_tn0.7: 81843
    87.0000 - val_fn0.7: 378614.0000 - val_precision0.7: 0.9500 - val_recall0.7: 0.7608 - val_tp0.9: 994204.0000 - val_fp0.9: 14550.0000 - val_tn0.9: 8233245.0000 - val_fn0.9: 588401.0000 - va
    l_precision0.9: 0.9856 - val_recall0.9: 0.6282 - val_accuracy: 0.9577 - val_auc: 0.9681 - val_f1: 0.2773
    Epoch 17/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2132 - tp0.1: 5740894.0000 - fp0.1: 2232275.0000 - tn0.1: 31152552.0000 - fn0.1: 195882.0000 - precision0.1: 0.7200 - rec
    all0.1: 0.9670 - tp0.3: 5532271.0000 - fp0.3: 1168226.0000 - tn0.3: 32216606.0000 - fn0.3: 404505.0000 - precision0.3: 0.8257 - recall0.3: 0.9319 - tp0.5: 5273881.0000 - fp0.5: 609565.0000
     - tn0.5: 32775244.0000 - fn0.5: 662895.0000 - precision0.5: 0.8964 - recall0.5: 0.8883 - tp0.7: 4954897.0000 - fp0.7: 286255.0000 - tn0.7: 33098576.0000 - fn0.7: 981879.0000 - precision0.
    7: 0.9454 - recall0.7: 0.8346 - tp0.9: 4392174.0000 - fp0.9: 78566.0000 - tn0.9: 33306278.0000 - fn0.9: 1544602.0000 - precision0.9: 0.9824 - recall0.9: 0.7398 - accuracy: 0.9676 - auc: 0.
    9792 - f1: 0.2624 - val_loss: 0.2674 - val_tp0.1: 1542015.0000 - val_fp0.1: 886833.0000 - val_tn0.1: 7360962.0000 - val_fn0.1: 40590.0000 - val_precision0.1: 0.6349 - val_recall0.1: 0.9744
     - val_tp0.3: 1513414.0000 - val_fp0.3: 533995.0000 - val_tn0.3: 7713800.0000 - val_fn0.3: 69191.0000 - val_precision0.3: 0.7392 - val_recall0.3: 0.9563 - val_tp0.5: 1469868.0000 - val_fp0
    .5: 320357.0000 - val_tn0.5: 7927438.0000 - val_fn0.5: 112737.0000 - val_precision0.5: 0.8211 - val_recall0.5: 0.9288 - val_tp0.7: 1389256.0000 - val_fp0.7: 180638.0000 - val_tn0.7: 806715
    7.0000 - val_fn0.7: 193349.0000 - val_precision0.7: 0.8849 - val_recall0.7: 0.8778 - val_tp0.9: 1265894.0000 - val_fp0.9: 72673.0000 - val_tn0.9: 8175122.0000 - val_fn0.9: 316711.0000 - va
    l_precision0.9: 0.9457 - val_recall0.9: 0.7999 - val_accuracy: 0.9559 - val_auc: 0.9797 - val_f1: 0.2773
    Epoch 18/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2127 - tp0.1: 5744925.0000 - fp0.1: 2262684.0000 - tn0.1: 31122132.0000 - fn0.1: 191851.0000 - precision0.1: 0.7174 - rec
    all0.1: 0.9677 - tp0.3: 5540864.0000 - fp0.3: 1151165.0000 - tn0.3: 32233664.0000 - fn0.3: 395912.0000 - precision0.3: 0.8280 - recall0.3: 0.9333 - tp0.5: 5275650.0000 - fp0.5: 595986.0000
     - tn0.5: 32788824.0000 - fn0.5: 661126.0000 - precision0.5: 0.8985 - recall0.5: 0.8886 - tp0.7: 4961721.0000 - fp0.7: 298376.0000 - tn0.7: 33086462.0000 - fn0.7: 975055.0000 - precision0.
    7: 0.9433 - recall0.7: 0.8358 - tp0.9: 4369101.0000 - fp0.9: 83285.0000 - tn0.9: 33301528.0000 - fn0.9: 1567675.0000 - precision0.9: 0.9813 - recall0.9: 0.7359 - accuracy: 0.9680 - auc: 0.
    9797 - f1: 0.2624 - val_loss: 0.2642 - val_tp0.1: 1498157.0000 - val_fp0.1: 419273.0000 - val_tn0.1: 7828522.0000 - val_fn0.1: 84448.0000 - val_precision0.1: 0.7813 - val_recall0.1: 0.9466
     - val_tp0.3: 1376765.0000 - val_fp0.3: 147925.0000 - val_tn0.3: 8099870.0000 - val_fn0.3: 205840.0000 - val_precision0.3: 0.9030 - val_recall0.3: 0.8699 - val_tp0.5: 1282934.0000 - val_fp
    0.5: 68148.0000 - val_tn0.5: 8179647.0000 - val_fn0.5: 299671.0000 - val_precision0.5: 0.9496 - val_recall0.5: 0.8106 - val_tp0.7: 1192758.0000 - val_fp0.7: 34480.0000 - val_tn0.7: 8213315
    .0000 - val_fn0.7: 389847.0000 - val_precision0.7: 0.9719 - val_recall0.7: 0.7537 - val_tp0.9: 940508.0000 - val_fp0.9: 5969.0000 - val_tn0.9: 8241826.0000 - val_fn0.9: 642097.0000 - val_p
    recision0.9: 0.9937 - val_recall0.9: 0.5943 - val_accuracy: 0.9626 - val_auc: 0.9698 - val_f1: 0.2773
    Epoch 19/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2024 - tp0.1: 5743078.0000 - fp0.1: 2054017.0000 - tn0.1: 31330796.0000 - fn0.1: 193698.0000 - precision0.1: 0.7366 - rec
    all0.1: 0.9674 - tp0.3: 5562740.0000 - fp0.3: 1087818.0000 - tn0.3: 32297020.0000 - fn0.3: 374036.0000 - precision0.3: 0.8364 - recall0.3: 0.9370 - tp0.5: 5320915.0000 - fp0.5: 581625.0000
     - tn0.5: 32803212.0000 - fn0.5: 615861.0000 - precision0.5: 0.9015 - recall0.5: 0.8963 - tp0.7: 5014211.0000 - fp0.7: 278329.0000 - tn0.7: 33106496.0000 - fn0.7: 922565.0000 - precision0.
    7: 0.9474 - recall0.7: 0.8446 - tp0.9: 4439455.0000 - fp0.9: 69924.0000 - tn0.9: 33314888.0000 - fn0.9: 1497321.0000 - precision0.9: 0.9845 - recall0.9: 0.7478 - accuracy: 0.9695 - auc: 0.
    9800 - f1: 0.2624 - val_loss: 0.2474 - val_tp0.1: 1500397.0000 - val_fp0.1: 388201.0000 - val_tn0.1: 7859594.0000 - val_fn0.1: 82208.0000 - val_precision0.1: 0.7945 - val_recall0.1: 0.9481
     - val_tp0.3: 1417832.0000 - val_fp0.3: 177501.0000 - val_tn0.3: 8070294.0000 - val_fn0.3: 164773.0000 - val_precision0.3: 0.8887 - val_recall0.3: 0.8959 - val_tp0.5: 1324841.0000 - val_fp
    0.5: 83172.0000 - val_tn0.5: 8164623.0000 - val_fn0.5: 257764.0000 - val_precision0.5: 0.9409 - val_recall0.5: 0.8371 - val_tp0.7: 1241641.0000 - val_fp0.7: 44194.0000 - val_tn0.7: 8203601
    .0000 - val_fn0.7: 340964.0000 - val_precision0.7: 0.9656 - val_recall0.7: 0.7846 - val_tp0.9: 1008204.0000 - val_fp0.9: 8267.0000 - val_tn0.9: 8239528.0000 - val_fn0.9: 574401.0000 - val_
    precision0.9: 0.9919 - val_recall0.9: 0.6371 - val_accuracy: 0.9653 - val_auc: 0.9703 - val_f1: 0.2773
    Epoch 20/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.1967 - tp0.1: 5751226.0000 - fp0.1: 2003963.0000 - tn0.1: 31380856.0000 - fn0.1: 185550.0000 - precision0.1: 0.7416 - rec
    all0.1: 0.9687 - tp0.3: 5579827.0000 - fp0.3: 1068977.0000 - tn0.3: 32315852.0000 - fn0.3: 356949.0000 - precision0.3: 0.8392 - recall0.3: 0.9399 - tp0.5: 5349645.0000 - fp0.5: 581039.0000
     - tn0.5: 32803788.0000 - fn0.5: 587131.0000 - precision0.5: 0.9020 - recall0.5: 0.9011 - tp0.7: 5051728.0000 - fp0.7: 282220.0000 - tn0.7: 33102590.0000 - fn0.7: 885048.0000 - precision0.
    7: 0.9471 - recall0.7: 0.8509 - tp0.9: 4482265.0000 - fp0.9: 74224.0000 - tn0.9: 33310598.0000 - fn0.9: 1454511.0000 - precision0.9: 0.9837 - recall0.9: 0.7550 - accuracy: 0.9703 - auc: 0.
    9808 - f1: 0.2624 - val_loss: 0.2503 - val_tp0.1: 1534044.0000 - val_fp0.1: 725110.0000 - val_tn0.1: 7522685.0000 - val_fn0.1: 48561.0000 - val_precision0.1: 0.6790 - val_recall0.1: 0.9693
     - val_tp0.3: 1480024.0000 - val_fp0.3: 353833.0000 - val_tn0.3: 7893962.0000 - val_fn0.3: 102581.0000 - val_precision0.3: 0.8071 - val_recall0.3: 0.9352 - val_tp0.5: 1407828.0000 - val_fp
    0.5: 193567.0000 - val_tn0.5: 8054228.0000 - val_fn0.5: 174777.0000 - val_precision0.5: 0.8791 - val_recall0.5: 0.8896 - val_tp0.7: 1340054.0000 - val_fp0.7: 119720.0000 - val_tn0.7: 81280
    75.0000 - val_fn0.7: 242551.0000 - val_precision0.7: 0.9180 - val_recall0.7: 0.8467 - val_tp0.9: 1179562.0000 - val_fp0.9: 41420.0000 - val_tn0.9: 8206375.0000 - val_fn0.9: 403043.0000 - v
    al_precision0.9: 0.9661 - val_recall0.9: 0.7453 - val_accuracy: 0.9625 - val_auc: 0.9780 - val_f1: 0.2773
    --- Running training session 56/140
    {'hp_epochs': 20, 'hp_batch_size': 10, 'hp_scaler': 'robust', 'hp_n_levels': 5, 'hp_first_filters': 16, 'hp_pool_size': 4, 'hp_input_size': 8192, 'hp_lr_start': 0.06276763366515732, 'hp_lr
    _power': 1.0}
    --- repeat #: 2
    input - shape:   (None, 8192, 1)
    output - shape:  (None, 8192, 1)
    Epoch 1/20
    480/480 [==============================] - 44s 52ms/step - loss: 0.6448 - tp0.1: 5265662.0000 - fp0.1: 6246252.0000 - tn0.1: 27138564.0000 - fn0.1: 671114.0000 - precision0.1: 0.4574 - rec
    all0.1: 0.8870 - tp0.3: 4710949.0000 - fp0.3: 3395106.0000 - tn0.3: 29989722.0000 - fn0.3: 1225827.0000 - precision0.3: 0.5812 - recall0.3: 0.7935 - tp0.5: 3693283.0000 - fp0.5: 1049113.00
    00 - tn0.5: 32335700.0000 - fn0.5: 2243493.0000 - precision0.5: 0.7788 - recall0.5: 0.6221 - tp0.7: 3063860.0000 - fp0.7: 405953.0000 - tn0.7: 32978848.0000 - fn0.7: 2872916.0000 - precisi
    on0.7: 0.8830 - recall0.7: 0.5161 - tp0.9: 2353054.0000 - fp0.9: 100482.0000 - tn0.9: 33284332.0000 - fn0.9: 3583722.0000 - precision0.9: 0.9590 - recall0.9: 0.3964 - accuracy: 0.9163 - au
    c: 0.9133 - f1: 0.2624 - val_loss: 1.3199 - val_tp0.1: 1131372.0000 - val_fp0.1: 737035.0000 - val_tn0.1: 7510760.0000 - val_fn0.1: 451233.0000 - val_precision0.1: 0.6055 - val_recall0.1:
    0.7149 - val_tp0.3: 1034683.0000 - val_fp0.3: 330216.0000 - val_tn0.3: 7917579.0000 - val_fn0.3: 547922.0000 - val_precision0.3: 0.7581 - val_recall0.3: 0.6538 - val_tp0.5: 928465.0000 - v
    al_fp0.5: 121376.0000 - val_tn0.5: 8126419.0000 - val_fn0.5: 654140.0000 - val_precision0.5: 0.8844 - val_recall0.5: 0.5867 - val_tp0.7: 829955.0000 - val_fp0.7: 60010.0000 - val_tn0.7: 81
    87785.0000 - val_fn0.7: 752650.0000 - val_precision0.7: 0.9326 - val_recall0.7: 0.5244 - val_tp0.9: 673776.0000 - val_fp0.9: 11090.0000 - val_tn0.9: 8236705.0000 - val_fn0.9: 908829.0000 -
     val_precision0.9: 0.9838 - val_recall0.9: 0.4257 - val_accuracy: 0.9211 - val_auc: 0.8400 - val_f1: 0.2773
    Epoch 2/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.5271 - tp0.1: 5418063.0000 - fp0.1: 5526907.0000 - tn0.1: 27857926.0000 - fn0.1: 518713.0000 - precision0.1: 0.4950 - rec
    all0.1: 0.9126 - tp0.3: 4862070.0000 - fp0.3: 2474103.0000 - tn0.3: 30910734.0000 - fn0.3: 1074706.0000 - precision0.3: 0.6628 - recall0.3: 0.8190 - tp0.5: 4124120.0000 - fp0.5: 1036855.00
    00 - tn0.5: 32347980.0000 - fn0.5: 1812656.0000 - precision0.5: 0.7991 - recall0.5: 0.6947 - tp0.7: 3446425.0000 - fp0.7: 437082.0000 - tn0.7: 32947768.0000 - fn0.7: 2490351.0000 - precisi
    on0.7: 0.8875 - recall0.7: 0.5805 - tp0.9: 2633706.0000 - fp0.9: 118948.0000 - tn0.9: 33265864.0000 - fn0.9: 3303070.0000 - precision0.9: 0.9568 - recall0.9: 0.4436 - accuracy: 0.9275 - au
    c: 0.9331 - f1: 0.2624 - val_loss: 1.1481 - val_tp0.1: 991975.0000 - val_fp0.1: 327918.0000 - val_tn0.1: 7919877.0000 - val_fn0.1: 590630.0000 - val_precision0.1: 0.7516 - val_recall0.1: 0
    .6268 - val_tp0.3: 921560.0000 - val_fp0.3: 203493.0000 - val_tn0.3: 8044302.0000 - val_fn0.3: 661045.0000 - val_precision0.3: 0.8191 - val_recall0.3: 0.5823 - val_tp0.5: 816746.0000 - val
    _fp0.5: 96370.0000 - val_tn0.5: 8151425.0000 - val_fn0.5: 765859.0000 - val_precision0.5: 0.8945 - val_recall0.5: 0.5161 - val_tp0.7: 638180.0000 - val_fp0.7: 16690.0000 - val_tn0.7: 82311
    05.0000 - val_fn0.7: 944425.0000 - val_precision0.7: 0.9745 - val_recall0.7: 0.4032 - val_tp0.9: 454222.0000 - val_fp0.9: 1300.0000 - val_tn0.9: 8246495.0000 - val_fn0.9: 1128383.0000 - va
    l_precision0.9: 0.9971 - val_recall0.9: 0.2870 - val_accuracy: 0.9123 - val_auc: 0.8046 - val_f1: 0.2773
    Epoch 3/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.4694 - tp0.1: 5506389.0000 - fp0.1: 5279352.0000 - tn0.1: 28105468.0000 - fn0.1: 430387.0000 - precision0.1: 0.5105 - rec
    all0.1: 0.9275 - tp0.3: 4935930.0000 - fp0.3: 2185008.0000 - tn0.3: 31199836.0000 - fn0.3: 1000846.0000 - precision0.3: 0.6932 - recall0.3: 0.8314 - tp0.5: 4402685.0000 - fp0.5: 1013988.00
    00 - tn0.5: 32370838.0000 - fn0.5: 1534091.0000 - precision0.5: 0.8128 - recall0.5: 0.7416 - tp0.7: 3775055.0000 - fp0.7: 446059.0000 - tn0.7: 32938768.0000 - fn0.7: 2161721.0000 - precisi
    on0.7: 0.8943 - recall0.7: 0.6359 - tp0.9: 2837543.0000 - fp0.9: 112976.0000 - tn0.9: 33271836.0000 - fn0.9: 3099233.0000 - precision0.9: 0.9617 - recall0.9: 0.4780 - accuracy: 0.9352 - au
    c: 0.9432 - f1: 0.2624 - val_loss: 1.1726 - val_tp0.1: 958675.0000 - val_fp0.1: 396142.0000 - val_tn0.1: 7851653.0000 - val_fn0.1: 623930.0000 - val_precision0.1: 0.7076 - val_recall0.1: 0
    .6058 - val_tp0.3: 888712.0000 - val_fp0.3: 224696.0000 - val_tn0.3: 8023099.0000 - val_fn0.3: 693893.0000 - val_precision0.3: 0.7982 - val_recall0.3: 0.5616 - val_tp0.5: 821633.0000 - val
    _fp0.5: 102326.0000 - val_tn0.5: 8145469.0000 - val_fn0.5: 760972.0000 - val_precision0.5: 0.8893 - val_recall0.5: 0.5192 - val_tp0.7: 735292.0000 - val_fp0.7: 39372.0000 - val_tn0.7: 8208
    423.0000 - val_fn0.7: 847313.0000 - val_precision0.7: 0.9492 - val_recall0.7: 0.4646 - val_tp0.9: 572482.0000 - val_fp0.9: 1157.0000 - val_tn0.9: 8246638.0000 - val_fn0.9: 1010123.0000 - v
    al_precision0.9: 0.9980 - val_recall0.9: 0.3617 - val_accuracy: 0.9122 - val_auc: 0.7943 - val_f1: 0.2773
    Epoch 4/20
    480/480 [==============================] - 21s 43ms/step - loss: 0.4326 - tp0.1: 5560029.0000 - fp0.1: 4926737.0000 - tn0.1: 28458088.0000 - fn0.1: 376747.0000 - precision0.1: 0.5302 - rec
    all0.1: 0.9365 - tp0.3: 5043034.0000 - fp0.3: 2165200.0000 - tn0.3: 31219610.0000 - fn0.3: 893742.0000 - precision0.3: 0.6996 - recall0.3: 0.8495 - tp0.5: 4510820.0000 - fp0.5: 990813.0000
     - tn0.5: 32394014.0000 - fn0.5: 1425956.0000 - precision0.5: 0.8199 - recall0.5: 0.7598 - tp0.7: 3929915.0000 - fp0.7: 460992.0000 - tn0.7: 32923836.0000 - fn0.7: 2006861.0000 - precision
    0.7: 0.8950 - recall0.7: 0.6620 - tp0.9: 2983151.0000 - fp0.9: 124501.0000 - tn0.9: 33260320.0000 - fn0.9: 2953625.0000 - precision0.9: 0.9599 - recall0.9: 0.5025 - accuracy: 0.9385 - auc:
     0.9491 - f1: 0.2624 - val_loss: 0.4556 - val_tp0.1: 1447920.0000 - val_fp0.1: 910577.0000 - val_tn0.1: 7337218.0000 - val_fn0.1: 134685.0000 - val_precision0.1: 0.6139 - val_recall0.1: 0.
    9149 - val_tp0.3: 1348323.0000 - val_fp0.3: 526913.0000 - val_tn0.3: 7720882.0000 - val_fn0.3: 234282.0000 - val_precision0.3: 0.7190 - val_recall0.3: 0.8520 - val_tp0.5: 1171829.0000 - va
    l_fp0.5: 254087.0000 - val_tn0.5: 7993708.0000 - val_fn0.5: 410776.0000 - val_precision0.5: 0.8218 - val_recall0.5: 0.7404 - val_tp0.7: 1027826.0000 - val_fp0.7: 124376.0000 - val_tn0.7: 8
    123419.0000 - val_fn0.7: 554779.0000 - val_precision0.7: 0.8921 - val_recall0.7: 0.6495 - val_tp0.9: 747344.0000 - val_fp0.9: 3170.0000 - val_tn0.9: 8244625.0000 - val_fn0.9: 835261.0000 -
     val_precision0.9: 0.9958 - val_recall0.9: 0.4722 - val_accuracy: 0.9324 - val_auc: 0.9418 - val_f1: 0.2773
    Epoch 5/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.4141 - tp0.1: 5574699.0000 - fp0.1: 4604743.0000 - tn0.1: 28780080.0000 - fn0.1: 362077.0000 - precision0.1: 0.5476 - rec
    all0.1: 0.9390 - tp0.3: 5097801.0000 - fp0.3: 2138526.0000 - tn0.3: 31246316.0000 - fn0.3: 838975.0000 - precision0.3: 0.7045 - recall0.3: 0.8587 - tp0.5: 4556087.0000 - fp0.5: 942923.0000
     - tn0.5: 32441904.0000 - fn0.5: 1380689.0000 - precision0.5: 0.8285 - recall0.5: 0.7674 - tp0.7: 3978370.0000 - fp0.7: 450803.0000 - tn0.7: 32934030.0000 - fn0.7: 1958406.0000 - precision
    0.7: 0.8982 - recall0.7: 0.6701 - tp0.9: 3000212.0000 - fp0.9: 125980.0000 - tn0.9: 33258844.0000 - fn0.9: 2936564.0000 - precision0.9: 0.9597 - recall0.9: 0.5054 - accuracy: 0.9409 - auc:
     0.9524 - f1: 0.2624 - val_loss: 0.4822 - val_tp0.1: 1476924.0000 - val_fp0.1: 1123072.0000 - val_tn0.1: 7124723.0000 - val_fn0.1: 105681.0000 - val_precision0.1: 0.5680 - val_recall0.1: 0
    .9332 - val_tp0.3: 1253583.0000 - val_fp0.3: 295322.0000 - val_tn0.3: 7952473.0000 - val_fn0.3: 329022.0000 - val_precision0.3: 0.8093 - val_recall0.3: 0.7921 - val_tp0.5: 1023581.0000 - v
    al_fp0.5: 50787.0000 - val_tn0.5: 8197008.0000 - val_fn0.5: 559024.0000 - val_precision0.5: 0.9527 - val_recall0.5: 0.6468 - val_tp0.7: 804131.0000 - val_fp0.7: 19006.0000 - val_tn0.7: 822
    8789.0000 - val_fn0.7: 778474.0000 - val_precision0.7: 0.9769 - val_recall0.7: 0.5081 - val_tp0.9: 623158.0000 - val_fp0.9: 3898.0000 - val_tn0.9: 8243897.0000 - val_fn0.9: 959447.0000 - v
    al_precision0.9: 0.9938 - val_recall0.9: 0.3938 - val_accuracy: 0.9380 - val_auc: 0.9508 - val_f1: 0.2773
    Epoch 6/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.3607 - tp0.1: 5623701.0000 - fp0.1: 3956775.0000 - tn0.1: 29428044.0000 - fn0.1: 313075.0000 - precision0.1: 0.5870 - rec
    all0.1: 0.9473 - tp0.3: 5201616.0000 - fp0.3: 1881599.0000 - tn0.3: 31503236.0000 - fn0.3: 735160.0000 - precision0.3: 0.7344 - recall0.3: 0.8762 - tp0.5: 4749347.0000 - fp0.5: 857493.0000
     - tn0.5: 32527328.0000 - fn0.5: 1187429.0000 - precision0.5: 0.8471 - recall0.5: 0.8000 - tp0.7: 4289257.0000 - fp0.7: 431585.0000 - tn0.7: 32953252.0000 - fn0.7: 1647519.0000 - precision
    0.7: 0.9086 - recall0.7: 0.7225 - tp0.9: 3310664.0000 - fp0.9: 120910.0000 - tn0.9: 33263930.0000 - fn0.9: 2626112.0000 - precision0.9: 0.9648 - recall0.9: 0.5577 - accuracy: 0.9480 - auc:
     0.9600 - f1: 0.2624 - val_loss: 0.3199 - val_tp0.1: 1487331.0000 - val_fp0.1: 694274.0000 - val_tn0.1: 7553521.0000 - val_fn0.1: 95274.0000 - val_precision0.1: 0.6818 - val_recall0.1: 0.9
    398 - val_tp0.3: 1386296.0000 - val_fp0.3: 286146.0000 - val_tn0.3: 7961649.0000 - val_fn0.3: 196309.0000 - val_precision0.3: 0.8289 - val_recall0.3: 0.8760 - val_tp0.5: 1304018.0000 - val
    _fp0.5: 139220.0000 - val_tn0.5: 8108575.0000 - val_fn0.5: 278587.0000 - val_precision0.5: 0.9035 - val_recall0.5: 0.8240 - val_tp0.7: 1235725.0000 - val_fp0.7: 76207.0000 - val_tn0.7: 817
    1588.0000 - val_fn0.7: 346880.0000 - val_precision0.7: 0.9419 - val_recall0.7: 0.7808 - val_tp0.9: 1080668.0000 - val_fp0.9: 23195.0000 - val_tn0.9: 8224600.0000 - val_fn0.9: 501937.0000 -
     val_precision0.9: 0.9790 - val_recall0.9: 0.6828 - val_accuracy: 0.9575 - val_auc: 0.9627 - val_f1: 0.2773
    Epoch 7/20
    480/480 [==============================] - 21s 43ms/step - loss: 0.3314 - tp0.1: 5646234.0000 - fp0.1: 3554455.0000 - tn0.1: 29830372.0000 - fn0.1: 290542.0000 - precision0.1: 0.6137 - rec
    all0.1: 0.9511 - tp0.3: 5227655.0000 - fp0.3: 1614171.0000 - tn0.3: 31770668.0000 - fn0.3: 709121.0000 - precision0.3: 0.7641 - recall0.3: 0.8806 - tp0.5: 4830553.0000 - fp0.5: 751013.0000
     - tn0.5: 32633820.0000 - fn0.5: 1106223.0000 - precision0.5: 0.8654 - recall0.5: 0.8137 - tp0.7: 4458898.0000 - fp0.7: 401272.0000 - tn0.7: 32983556.0000 - fn0.7: 1477878.0000 - precision
    0.7: 0.9174 - recall0.7: 0.7511 - tp0.9: 3589678.0000 - fp0.9: 117919.0000 - tn0.9: 33266924.0000 - fn0.9: 2347098.0000 - precision0.9: 0.9682 - recall0.9: 0.6047 - accuracy: 0.9528 - auc:
     0.9641 - f1: 0.2624 - val_loss: 0.3516 - val_tp0.1: 1443246.0000 - val_fp0.1: 388491.0000 - val_tn0.1: 7859304.0000 - val_fn0.1: 139359.0000 - val_precision0.1: 0.7879 - val_recall0.1: 0.
    9119 - val_tp0.3: 1363949.0000 - val_fp0.3: 220386.0000 - val_tn0.3: 8027409.0000 - val_fn0.3: 218656.0000 - val_precision0.3: 0.8609 - val_recall0.3: 0.8618 - val_tp0.5: 1277387.0000 - va
    l_fp0.5: 127462.0000 - val_tn0.5: 8120333.0000 - val_fn0.5: 305218.0000 - val_precision0.5: 0.9093 - val_recall0.5: 0.8071 - val_tp0.7: 1201550.0000 - val_fp0.7: 76391.0000 - val_tn0.7: 81
    71404.0000 - val_fn0.7: 381055.0000 - val_precision0.7: 0.9402 - val_recall0.7: 0.7592 - val_tp0.9: 1034837.0000 - val_fp0.9: 29131.0000 - val_tn0.9: 8218664.0000 - val_fn0.9: 547768.0000
    - val_precision0.9: 0.9726 - val_recall0.9: 0.6539 - val_accuracy: 0.9560 - val_auc: 0.9517 - val_f1: 0.2773
    Epoch 8/20
    480/480 [==============================] - 21s 43ms/step - loss: 0.3124 - tp0.1: 5661924.0000 - fp0.1: 3284244.0000 - tn0.1: 30100560.0000 - fn0.1: 274852.0000 - precision0.1: 0.6329 - rec
    all0.1: 0.9537 - tp0.3: 5341190.0000 - fp0.3: 1750123.0000 - tn0.3: 31634684.0000 - fn0.3: 595586.0000 - precision0.3: 0.7532 - recall0.3: 0.8997 - tp0.5: 4855283.0000 - fp0.5: 738314.0000
     - tn0.5: 32646522.0000 - fn0.5: 1081493.0000 - precision0.5: 0.8680 - recall0.5: 0.8178 - tp0.7: 4484249.0000 - fp0.7: 391982.0000 - tn0.7: 32992836.0000 - fn0.7: 1452527.0000 - precision
    0.7: 0.9196 - recall0.7: 0.7553 - tp0.9: 3662123.0000 - fp0.9: 116143.0000 - tn0.9: 33268688.0000 - fn0.9: 2274653.0000 - precision0.9: 0.9693 - recall0.9: 0.6169 - accuracy: 0.9537 - auc:
     0.9668 - f1: 0.2624 - val_loss: 0.3804 - val_tp0.1: 1538470.0000 - val_fp0.1: 1344786.0000 - val_tn0.1: 6903009.0000 - val_fn0.1: 44135.0000 - val_precision0.1: 0.5336 - val_recall0.1: 0.
    9721 - val_tp0.3: 1492276.0000 - val_fp0.3: 750571.0000 - val_tn0.3: 7497224.0000 - val_fn0.3: 90329.0000 - val_precision0.3: 0.6653 - val_recall0.3: 0.9429 - val_tp0.5: 1415057.0000 - val
    _fp0.5: 403390.0000 - val_tn0.5: 7844405.0000 - val_fn0.5: 167548.0000 - val_precision0.5: 0.7782 - val_recall0.5: 0.8941 - val_tp0.7: 1355676.0000 - val_fp0.7: 251522.0000 - val_tn0.7: 79
    96273.0000 - val_fn0.7: 226929.0000 - val_precision0.7: 0.8435 - val_recall0.7: 0.8566 - val_tp0.9: 1239605.0000 - val_fp0.9: 115484.0000 - val_tn0.9: 8132311.0000 - val_fn0.9: 343000.0000
     - val_precision0.9: 0.9148 - val_recall0.9: 0.7833 - val_accuracy: 0.9419 - val_auc: 0.9731 - val_f1: 0.2773
    Epoch 9/20
    480/480 [==============================] - 21s 43ms/step - loss: 0.3173 - tp0.1: 5659382.0000 - fp0.1: 3425500.0000 - tn0.1: 29959316.0000 - fn0.1: 277394.0000 - precision0.1: 0.6229 - rec
    all0.1: 0.9533 - tp0.3: 5292646.0000 - fp0.3: 1671784.0000 - tn0.3: 31713034.0000 - fn0.3: 644130.0000 - precision0.3: 0.7600 - recall0.3: 0.8915 - tp0.5: 4877061.0000 - fp0.5: 747610.0000
     - tn0.5: 32637216.0000 - fn0.5: 1059715.0000 - precision0.5: 0.8671 - recall0.5: 0.8215 - tp0.7: 4496117.0000 - fp0.7: 402814.0000 - tn0.7: 32982008.0000 - fn0.7: 1440659.0000 - precision
    0.7: 0.9178 - recall0.7: 0.7573 - tp0.9: 3666370.0000 - fp0.9: 116952.0000 - tn0.9: 33267872.0000 - fn0.9: 2270406.0000 - precision0.9: 0.9691 - recall0.9: 0.6176 - accuracy: 0.9540 - auc:
     0.9663 - f1: 0.2624 - val_loss: 0.4111 - val_tp0.1: 1461116.0000 - val_fp0.1: 755222.0000 - val_tn0.1: 7492573.0000 - val_fn0.1: 121489.0000 - val_precision0.1: 0.6592 - val_recall0.1: 0.
    9232 - val_tp0.3: 1223139.0000 - val_fp0.3: 230361.0000 - val_tn0.3: 8017434.0000 - val_fn0.3: 359466.0000 - val_precision0.3: 0.8415 - val_recall0.3: 0.7729 - val_tp0.5: 1066667.0000 - va
    l_fp0.5: 102255.0000 - val_tn0.5: 8145540.0000 - val_fn0.5: 515938.0000 - val_precision0.5: 0.9125 - val_recall0.5: 0.6740 - val_tp0.7: 886969.0000 - val_fp0.7: 41575.0000 - val_tn0.7: 820
    6220.0000 - val_fn0.7: 695636.0000 - val_precision0.7: 0.9552 - val_recall0.7: 0.5604 - val_tp0.9: 665393.0000 - val_fp0.9: 8132.0000 - val_tn0.9: 8239663.0000 - val_fn0.9: 917212.0000 - v
    al_precision0.9: 0.9879 - val_recall0.9: 0.4204 - val_accuracy: 0.9371 - val_auc: 0.9500 - val_f1: 0.2773
    Epoch 10/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2749 - tp0.1: 5674290.0000 - fp0.1: 2738682.0000 - tn0.1: 30646168.0000 - fn0.1: 262486.0000 - precision0.1: 0.6745 - rec
    all0.1: 0.9558 - tp0.3: 5383213.0000 - fp0.3: 1364871.0000 - tn0.3: 32019970.0000 - fn0.3: 553563.0000 - precision0.3: 0.7977 - recall0.3: 0.9068 - tp0.5: 4998441.0000 - fp0.5: 611762.0000
     - tn0.5: 32773056.0000 - fn0.5: 938335.0000 - precision0.5: 0.8910 - recall0.5: 0.8419 - tp0.7: 4697294.0000 - fp0.7: 334198.0000 - tn0.7: 33050624.0000 - fn0.7: 1239482.0000 - precision0
    .7: 0.9336 - recall0.7: 0.7912 - tp0.9: 4025561.0000 - fp0.9: 94510.0000 - tn0.9: 33290300.0000 - fn0.9: 1911215.0000 - precision0.9: 0.9771 - recall0.9: 0.6781 - accuracy: 0.9606 - auc: 0
    .9710 - f1: 0.2624 - val_loss: 0.3147 - val_tp0.1: 1502641.0000 - val_fp0.1: 644566.0000 - val_tn0.1: 7603229.0000 - val_fn0.1: 79964.0000 - val_precision0.1: 0.6998 - val_recall0.1: 0.949
    5 - val_tp0.3: 1470125.0000 - val_fp0.3: 446059.0000 - val_tn0.3: 7801736.0000 - val_fn0.3: 112480.0000 - val_precision0.3: 0.7672 - val_recall0.3: 0.9289 - val_tp0.5: 1426341.0000 - val_f
    p0.5: 321683.0000 - val_tn0.5: 7926112.0000 - val_fn0.5: 156264.0000 - val_precision0.5: 0.8160 - val_recall0.5: 0.9013 - val_tp0.7: 1390033.0000 - val_fp0.7: 241453.0000 - val_tn0.7: 8006
    342.0000 - val_fn0.7: 192572.0000 - val_precision0.7: 0.8520 - val_recall0.7: 0.8783 - val_tp0.9: 1306070.0000 - val_fp0.9: 125298.0000 - val_tn0.9: 8122497.0000 - val_fn0.9: 276535.0000 -
     val_precision0.9: 0.9125 - val_recall0.9: 0.8253 - val_accuracy: 0.9514 - val_auc: 0.9668 - val_f1: 0.2773
    Epoch 11/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2571 - tp0.1: 5701636.0000 - fp0.1: 2654470.0000 - tn0.1: 30730342.0000 - fn0.1: 235140.0000 - precision0.1: 0.6823 - rec
    all0.1: 0.9604 - tp0.3: 5431401.0000 - fp0.3: 1345161.0000 - tn0.3: 32039650.0000 - fn0.3: 505375.0000 - precision0.3: 0.8015 - recall0.3: 0.9149 - tp0.5: 5052982.0000 - fp0.5: 600086.0000
     - tn0.5: 32784728.0000 - fn0.5: 883794.0000 - precision0.5: 0.8938 - recall0.5: 0.8511 - tp0.7: 4767820.0000 - fp0.7: 330196.0000 - tn0.7: 33054604.0000 - fn0.7: 1168956.0000 - precision0
    .7: 0.9352 - recall0.7: 0.8031 - tp0.9: 4129243.0000 - fp0.9: 91729.0000 - tn0.9: 33293112.0000 - fn0.9: 1807533.0000 - precision0.9: 0.9783 - recall0.9: 0.6955 - accuracy: 0.9623 - auc: 0
    .9739 - f1: 0.2624 - val_loss: 0.2934 - val_tp0.1: 1497012.0000 - val_fp0.1: 580336.0000 - val_tn0.1: 7667459.0000 - val_fn0.1: 85593.0000 - val_precision0.1: 0.7206 - val_recall0.1: 0.945
    9 - val_tp0.3: 1418177.0000 - val_fp0.3: 304844.0000 - val_tn0.3: 7942951.0000 - val_fn0.3: 164428.0000 - val_precision0.3: 0.8231 - val_recall0.3: 0.8961 - val_tp0.5: 1344607.0000 - val_f
    p0.5: 177499.0000 - val_tn0.5: 8070296.0000 - val_fn0.5: 237998.0000 - val_precision0.5: 0.8834 - val_recall0.5: 0.8496 - val_tp0.7: 1277773.0000 - val_fp0.7: 107504.0000 - val_tn0.7: 8140
    291.0000 - val_fn0.7: 304832.0000 - val_precision0.7: 0.9224 - val_recall0.7: 0.8074 - val_tp0.9: 1107705.0000 - val_fp0.9: 32564.0000 - val_tn0.9: 8215231.0000 - val_fn0.9: 474900.0000 -
    val_precision0.9: 0.9714 - val_recall0.9: 0.6999 - val_accuracy: 0.9577 - val_auc: 0.9663 - val_f1: 0.2773
    Epoch 12/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2581 - tp0.1: 5693123.0000 - fp0.1: 2595417.0000 - tn0.1: 30789420.0000 - fn0.1: 243653.0000 - precision0.1: 0.6869 - rec
    all0.1: 0.9590 - tp0.3: 5446545.0000 - fp0.3: 1373916.0000 - tn0.3: 32010920.0000 - fn0.3: 490231.0000 - precision0.3: 0.7986 - recall0.3: 0.9174 - tp0.5: 5065870.0000 - fp0.5: 607551.0000
     - tn0.5: 32777282.0000 - fn0.5: 870906.0000 - precision0.5: 0.8929 - recall0.5: 0.8533 - tp0.7: 4775092.0000 - fp0.7: 334512.0000 - tn0.7: 33050316.0000 - fn0.7: 1161684.0000 - precision0
    .7: 0.9345 - recall0.7: 0.8043 - tp0.9: 4147063.0000 - fp0.9: 95688.0000 - tn0.9: 33289134.0000 - fn0.9: 1789713.0000 - precision0.9: 0.9774 - recall0.9: 0.6985 - accuracy: 0.9624 - auc: 0
    .9732 - f1: 0.2624 - val_loss: 0.3343 - val_tp0.1: 1442029.0000 - val_fp0.1: 269201.0000 - val_tn0.1: 7978594.0000 - val_fn0.1: 140576.0000 - val_precision0.1: 0.8427 - val_recall0.1: 0.91
    12 - val_tp0.3: 1407074.0000 - val_fp0.3: 174843.0000 - val_tn0.3: 8072952.0000 - val_fn0.3: 175531.0000 - val_precision0.3: 0.8895 - val_recall0.3: 0.8891 - val_tp0.5: 1303677.0000 - val_
    fp0.5: 75577.0000 - val_tn0.5: 8172218.0000 - val_fn0.5: 278928.0000 - val_precision0.5: 0.9452 - val_recall0.5: 0.8238 - val_tp0.7: 1215800.0000 - val_fp0.7: 37337.0000 - val_tn0.7: 82104
    58.0000 - val_fn0.7: 366805.0000 - val_precision0.7: 0.9702 - val_recall0.7: 0.7682 - val_tp0.9: 998956.0000 - val_fp0.9: 6587.0000 - val_tn0.9: 8241208.0000 - val_fn0.9: 583649.0000 - val
    _precision0.9: 0.9934 - val_recall0.9: 0.6312 - val_accuracy: 0.9639 - val_auc: 0.9540 - val_f1: 0.2773
    Epoch 13/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2406 - tp0.1: 5706933.0000 - fp0.1: 2415054.0000 - tn0.1: 30969764.0000 - fn0.1: 229843.0000 - precision0.1: 0.7027 - rec
    all0.1: 0.9613 - tp0.3: 5471726.0000 - fp0.3: 1216004.0000 - tn0.3: 32168800.0000 - fn0.3: 465050.0000 - precision0.3: 0.8182 - recall0.3: 0.9217 - tp0.5: 5159436.0000 - fp0.5: 606629.0000
     - tn0.5: 32778208.0000 - fn0.5: 777340.0000 - precision0.5: 0.8948 - recall0.5: 0.8691 - tp0.7: 4865465.0000 - fp0.7: 318865.0000 - tn0.7: 33065960.0000 - fn0.7: 1071311.0000 - precision0
    .7: 0.9385 - recall0.7: 0.8195 - tp0.9: 4237026.0000 - fp0.9: 90188.0000 - tn0.9: 33294624.0000 - fn0.9: 1699750.0000 - precision0.9: 0.9792 - recall0.9: 0.7137 - accuracy: 0.9648 - auc: 0
    .9754 - f1: 0.2624 - val_loss: 0.2832 - val_tp0.1: 1508198.0000 - val_fp0.1: 679566.0000 - val_tn0.1: 7568229.0000 - val_fn0.1: 74407.0000 - val_precision0.1: 0.6894 - val_recall0.1: 0.953
    0 - val_tp0.3: 1446107.0000 - val_fp0.3: 334069.0000 - val_tn0.3: 7913726.0000 - val_fn0.3: 136498.0000 - val_precision0.3: 0.8123 - val_recall0.3: 0.9138 - val_tp0.5: 1377613.0000 - val_f
    p0.5: 188915.0000 - val_tn0.5: 8058880.0000 - val_fn0.5: 204992.0000 - val_precision0.5: 0.8794 - val_recall0.5: 0.8705 - val_tp0.7: 1285384.0000 - val_fp0.7: 100950.0000 - val_tn0.7: 8146
    845.0000 - val_fn0.7: 297221.0000 - val_precision0.7: 0.9272 - val_recall0.7: 0.8122 - val_tp0.9: 1070502.0000 - val_fp0.9: 23679.0000 - val_tn0.9: 8224116.0000 - val_fn0.9: 512103.0000 -
    val_precision0.9: 0.9784 - val_recall0.9: 0.6764 - val_accuracy: 0.9599 - val_auc: 0.9699 - val_f1: 0.2773
    Epoch 14/20
    480/480 [==============================] - 21s 43ms/step - loss: 0.2360 - tp0.1: 5707005.0000 - fp0.1: 2311718.0000 - tn0.1: 31073108.0000 - fn0.1: 229771.0000 - precision0.1: 0.7117 - rec
    all0.1: 0.9613 - tp0.3: 5487802.0000 - fp0.3: 1178541.0000 - tn0.3: 32206284.0000 - fn0.3: 448974.0000 - precision0.3: 0.8232 - recall0.3: 0.9244 - tp0.5: 5194613.0000 - fp0.5: 583262.0000
     - tn0.5: 32801574.0000 - fn0.5: 742163.0000 - precision0.5: 0.8991 - recall0.5: 0.8750 - tp0.7: 4910863.0000 - fp0.7: 316507.0000 - tn0.7: 33068318.0000 - fn0.7: 1025913.0000 - precision0
    .7: 0.9395 - recall0.7: 0.8272 - tp0.9: 4295202.0000 - fp0.9: 88374.0000 - tn0.9: 33296440.0000 - fn0.9: 1641574.0000 - precision0.9: 0.9798 - recall0.9: 0.7235 - accuracy: 0.9663 - auc: 0
    .9761 - f1: 0.2624 - val_loss: 0.2611 - val_tp0.1: 1505124.0000 - val_fp0.1: 555660.0000 - val_tn0.1: 7692135.0000 - val_fn0.1: 77481.0000 - val_precision0.1: 0.7304 - val_recall0.1: 0.951
    0 - val_tp0.3: 1441677.0000 - val_fp0.3: 299314.0000 - val_tn0.3: 7948481.0000 - val_fn0.3: 140928.0000 - val_precision0.3: 0.8281 - val_recall0.3: 0.9110 - val_tp0.5: 1368785.0000 - val_f
    p0.5: 147030.0000 - val_tn0.5: 8100765.0000 - val_fn0.5: 213820.0000 - val_precision0.5: 0.9030 - val_recall0.5: 0.8649 - val_tp0.7: 1283194.0000 - val_fp0.7: 69148.0000 - val_tn0.7: 81786
    47.0000 - val_fn0.7: 299411.0000 - val_precision0.7: 0.9489 - val_recall0.7: 0.8108 - val_tp0.9: 1099339.0000 - val_fp0.9: 14278.0000 - val_tn0.9: 8233517.0000 - val_fn0.9: 483266.0000 - v
    al_precision0.9: 0.9872 - val_recall0.9: 0.6946 - val_accuracy: 0.9633 - val_auc: 0.9706 - val_f1: 0.2773
    Epoch 15/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2284 - tp0.1: 5723340.0000 - fp0.1: 2332080.0000 - tn0.1: 31052740.0000 - fn0.1: 213436.0000 - precision0.1: 0.7105 - rec
    all0.1: 0.9640 - tp0.3: 5500588.0000 - fp0.3: 1197957.0000 - tn0.3: 32186864.0000 - fn0.3: 436188.0000 - precision0.3: 0.8212 - recall0.3: 0.9265 - tp0.5: 5193950.0000 - fp0.5: 592030.0000
     - tn0.5: 32792800.0000 - fn0.5: 742826.0000 - precision0.5: 0.8977 - recall0.5: 0.8749 - tp0.7: 4891648.0000 - fp0.7: 303851.0000 - tn0.7: 33080976.0000 - fn0.7: 1045128.0000 - precision0
    .7: 0.9415 - recall0.7: 0.8240 - tp0.9: 4289986.0000 - fp0.9: 85261.0000 - tn0.9: 33299542.0000 - fn0.9: 1646790.0000 - precision0.9: 0.9805 - recall0.9: 0.7226 - accuracy: 0.9661 - auc: 0
    .9772 - f1: 0.2624 - val_loss: 0.2553 - val_tp0.1: 1513456.0000 - val_fp0.1: 572837.0000 - val_tn0.1: 7674958.0000 - val_fn0.1: 69149.0000 - val_precision0.1: 0.7254 - val_recall0.1: 0.956
    3 - val_tp0.3: 1455197.0000 - val_fp0.3: 304127.0000 - val_tn0.3: 7943668.0000 - val_fn0.3: 127408.0000 - val_precision0.3: 0.8271 - val_recall0.3: 0.9195 - val_tp0.5: 1393774.0000 - val_f
    p0.5: 176303.0000 - val_tn0.5: 8071492.0000 - val_fn0.5: 188831.0000 - val_precision0.5: 0.8877 - val_recall0.5: 0.8807 - val_tp0.7: 1315535.0000 - val_fp0.7: 92255.0000 - val_tn0.7: 81555
    40.0000 - val_fn0.7: 267070.0000 - val_precision0.7: 0.9345 - val_recall0.7: 0.8312 - val_tp0.9: 1145098.0000 - val_fp0.9: 24287.0000 - val_tn0.9: 8223508.0000 - val_fn0.9: 437507.0000 - v
    al_precision0.9: 0.9792 - val_recall0.9: 0.7236 - val_accuracy: 0.9629 - val_auc: 0.9728 - val_f1: 0.2773
    Epoch 16/20
    480/480 [==============================] - 21s 43ms/step - loss: 0.2187 - tp0.1: 5730057.0000 - fp0.1: 2238043.0000 - tn0.1: 31146768.0000 - fn0.1: 206719.0000 - precision0.1: 0.7191 - rec
    all0.1: 0.9652 - tp0.3: 5526887.0000 - fp0.3: 1168477.0000 - tn0.3: 32216352.0000 - fn0.3: 409889.0000 - precision0.3: 0.8255 - recall0.3: 0.9310 - tp0.5: 5240624.0000 - fp0.5: 596087.0000
     - tn0.5: 32788744.0000 - fn0.5: 696152.0000 - precision0.5: 0.8979 - recall0.5: 0.8827 - tp0.7: 4938991.0000 - fp0.7: 293175.0000 - tn0.7: 33091644.0000 - fn0.7: 997785.0000 - precision0.
    7: 0.9440 - recall0.7: 0.8319 - tp0.9: 4369873.0000 - fp0.9: 82397.0000 - tn0.9: 33302432.0000 - fn0.9: 1566903.0000 - precision0.9: 0.9815 - recall0.9: 0.7361 - accuracy: 0.9671 - auc: 0.
    9784 - f1: 0.2624 - val_loss: 0.2767 - val_tp0.1: 1523233.0000 - val_fp0.1: 780683.0000 - val_tn0.1: 7467112.0000 - val_fn0.1: 59372.0000 - val_precision0.1: 0.6611 - val_recall0.1: 0.9625
     - val_tp0.3: 1444414.0000 - val_fp0.3: 310339.0000 - val_tn0.3: 7937456.0000 - val_fn0.3: 138191.0000 - val_precision0.3: 0.8231 - val_recall0.3: 0.9127 - val_tp0.5: 1372018.0000 - val_fp
    0.5: 162131.0000 - val_tn0.5: 8085664.0000 - val_fn0.5: 210587.0000 - val_precision0.5: 0.8943 - val_recall0.5: 0.8669 - val_tp0.7: 1293148.0000 - val_fp0.7: 87275.0000 - val_tn0.7: 816052
    0.0000 - val_fn0.7: 289457.0000 - val_precision0.7: 0.9368 - val_recall0.7: 0.8171 - val_tp0.9: 1123318.0000 - val_fp0.9: 21366.0000 - val_tn0.9: 8226429.0000 - val_fn0.9: 459287.0000 - va
    l_precision0.9: 0.9813 - val_recall0.9: 0.7098 - val_accuracy: 0.9621 - val_auc: 0.9750 - val_f1: 0.2773
    Epoch 17/20
    480/480 [==============================] - 21s 43ms/step - loss: 0.2057 - tp0.1: 5743364.0000 - fp0.1: 2092169.0000 - tn0.1: 31292644.0000 - fn0.1: 193412.0000 - precision0.1: 0.7330 - rec
    all0.1: 0.9674 - tp0.3: 5554900.0000 - fp0.3: 1112642.0000 - tn0.3: 32272176.0000 - fn0.3: 381876.0000 - precision0.3: 0.8331 - recall0.3: 0.9357 - tp0.5: 5313778.0000 - fp0.5: 596612.0000
     - tn0.5: 32788216.0000 - fn0.5: 622998.0000 - precision0.5: 0.8991 - recall0.5: 0.8951 - tp0.7: 4988960.0000 - fp0.7: 279744.0000 - tn0.7: 33105068.0000 - fn0.7: 947816.0000 - precision0.
    7: 0.9469 - recall0.7: 0.8403 - tp0.9: 4440493.0000 - fp0.9: 78516.0000 - tn0.9: 33306316.0000 - fn0.9: 1496283.0000 - precision0.9: 0.9826 - recall0.9: 0.7480 - accuracy: 0.9690 - auc: 0.
    9798 - f1: 0.2624 - val_loss: 0.2385 - val_tp0.1: 1523932.0000 - val_fp0.1: 579099.0000 - val_tn0.1: 7668696.0000 - val_fn0.1: 58673.0000 - val_precision0.1: 0.7246 - val_recall0.1: 0.9629
     - val_tp0.3: 1459850.0000 - val_fp0.3: 271056.0000 - val_tn0.3: 7976739.0000 - val_fn0.3: 122755.0000 - val_precision0.3: 0.8434 - val_recall0.3: 0.9224 - val_tp0.5: 1390400.0000 - val_fp
    0.5: 143841.0000 - val_tn0.5: 8103954.0000 - val_fn0.5: 192205.0000 - val_precision0.5: 0.9062 - val_recall0.5: 0.8786 - val_tp0.7: 1305162.0000 - val_fp0.7: 75449.0000 - val_tn0.7: 817234
    6.0000 - val_fn0.7: 277443.0000 - val_precision0.7: 0.9454 - val_recall0.7: 0.8247 - val_tp0.9: 1149169.0000 - val_fp0.9: 23232.0000 - val_tn0.9: 8224563.0000 - val_fn0.9: 433436.0000 - va
    l_precision0.9: 0.9802 - val_recall0.9: 0.7261 - val_accuracy: 0.9658 - val_auc: 0.9764 - val_f1: 0.2773
    Epoch 18/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.2093 - tp0.1: 5739105.0000 - fp0.1: 2145054.0000 - tn0.1: 31239784.0000 - fn0.1: 197671.0000 - precision0.1: 0.7279 - rec
    all0.1: 0.9667 - tp0.3: 5545657.0000 - fp0.3: 1100602.0000 - tn0.3: 32284224.0000 - fn0.3: 391119.0000 - precision0.3: 0.8344 - recall0.3: 0.9341 - tp0.5: 5306649.0000 - fp0.5: 593735.0000
     - tn0.5: 32791100.0000 - fn0.5: 630127.0000 - precision0.5: 0.8994 - recall0.5: 0.8939 - tp0.7: 4982136.0000 - fp0.7: 287791.0000 - tn0.7: 33097036.0000 - fn0.7: 954640.0000 - precision0.
    7: 0.9454 - recall0.7: 0.8392 - tp0.9: 4415070.0000 - fp0.9: 79761.0000 - tn0.9: 33305060.0000 - fn0.9: 1521706.0000 - precision0.9: 0.9823 - recall0.9: 0.7437 - accuracy: 0.9689 - auc: 0.
    9796 - f1: 0.2624 - val_loss: 0.2350 - val_tp0.1: 1525598.0000 - val_fp0.1: 568429.0000 - val_tn0.1: 7679366.0000 - val_fn0.1: 57007.0000 - val_precision0.1: 0.7285 - val_recall0.1: 0.9640
     - val_tp0.3: 1474346.0000 - val_fp0.3: 295141.0000 - val_tn0.3: 7952654.0000 - val_fn0.3: 108259.0000 - val_precision0.3: 0.8332 - val_recall0.3: 0.9316 - val_tp0.5: 1424497.0000 - val_fp
    0.5: 190561.0000 - val_tn0.5: 8057234.0000 - val_fn0.5: 158108.0000 - val_precision0.5: 0.8820 - val_recall0.5: 0.9001 - val_tp0.7: 1359738.0000 - val_fp0.7: 110514.0000 - val_tn0.7: 81372
    81.0000 - val_fn0.7: 222867.0000 - val_precision0.7: 0.9248 - val_recall0.7: 0.8592 - val_tp0.9: 1242403.0000 - val_fp0.9: 43687.0000 - val_tn0.9: 8204108.0000 - val_fn0.9: 340202.0000 - v
    al_precision0.9: 0.9660 - val_recall0.9: 0.7850 - val_accuracy: 0.9645 - val_auc: 0.9772 - val_f1: 0.2773
    Epoch 19/20
    480/480 [==============================] - 21s 43ms/step - loss: 0.1984 - tp0.1: 5744440.0000 - fp0.1: 1989366.0000 - tn0.1: 31395464.0000 - fn0.1: 192336.0000 - precision0.1: 0.7428 - rec
    all0.1: 0.9676 - tp0.3: 5566674.0000 - fp0.3: 1040758.0000 - tn0.3: 32344072.0000 - fn0.3: 370102.0000 - precision0.3: 0.8425 - recall0.3: 0.9377 - tp0.5: 5348217.0000 - fp0.5: 577516.0000
     - tn0.5: 32807312.0000 - fn0.5: 588559.0000 - precision0.5: 0.9025 - recall0.5: 0.9009 - tp0.7: 5052821.0000 - fp0.7: 276996.0000 - tn0.7: 33107824.0000 - fn0.7: 883955.0000 - precision0.
    7: 0.9480 - recall0.7: 0.8511 - tp0.9: 4492968.0000 - fp0.9: 72302.0000 - tn0.9: 33312520.0000 - fn0.9: 1443808.0000 - precision0.9: 0.9842 - recall0.9: 0.7568 - accuracy: 0.9703 - auc: 0.
    9805 - f1: 0.2624 - val_loss: 0.2484 - val_tp0.1: 1501131.0000 - val_fp0.1: 393771.0000 - val_tn0.1: 7854024.0000 - val_fn0.1: 81474.0000 - val_precision0.1: 0.7922 - val_recall0.1: 0.9485
     - val_tp0.3: 1409758.0000 - val_fp0.3: 163787.0000 - val_tn0.3: 8084008.0000 - val_fn0.3: 172847.0000 - val_precision0.3: 0.8959 - val_recall0.3: 0.8908 - val_tp0.5: 1322183.0000 - val_fp
    0.5: 84259.0000 - val_tn0.5: 8163536.0000 - val_fn0.5: 260422.0000 - val_precision0.5: 0.9401 - val_recall0.5: 0.8354 - val_tp0.7: 1216424.0000 - val_fp0.7: 42192.0000 - val_tn0.7: 8205603
    .0000 - val_fn0.7: 366181.0000 - val_precision0.7: 0.9665 - val_recall0.7: 0.7686 - val_tp0.9: 1024424.0000 - val_fp0.9: 10587.0000 - val_tn0.9: 8237208.0000 - val_fn0.9: 558181.0000 - val
    _precision0.9: 0.9898 - val_recall0.9: 0.6473 - val_accuracy: 0.9649 - val_auc: 0.9712 - val_f1: 0.2773
    Epoch 20/20
    480/480 [==============================] - 21s 44ms/step - loss: 0.1942 - tp0.1: 5753146.0000 - fp0.1: 1975562.0000 - tn0.1: 31409254.0000 - fn0.1: 183630.0000 - precision0.1: 0.7444 - rec
    all0.1: 0.9691 - tp0.3: 5571490.0000 - fp0.3: 1010543.0000 - tn0.3: 32374284.0000 - fn0.3: 365286.0000 - precision0.3: 0.8465 - recall0.3: 0.9385 - tp0.5: 5341643.0000 - fp0.5: 549462.0000
     - tn0.5: 32835368.0000 - fn0.5: 595133.0000 - precision0.5: 0.9067 - recall0.5: 0.8998 - tp0.7: 5040559.0000 - fp0.7: 265388.0000 - tn0.7: 33119436.0000 - fn0.7: 896217.0000 - precision0.
    7: 0.9500 - recall0.7: 0.8490 - tp0.9: 4493527.0000 - fp0.9: 71308.0000 - tn0.9: 33313516.0000 - fn0.9: 1443249.0000 - precision0.9: 0.9844 - recall0.9: 0.7569 - accuracy: 0.9709 - auc: 0.
    9814 - f1: 0.2624 - val_loss: 0.2216 - val_tp0.1: 1527498.0000 - val_fp0.1: 536072.0000 - val_tn0.1: 7711723.0000 - val_fn0.1: 55107.0000 - val_precision0.1: 0.7402 - val_recall0.1: 0.9652
     - val_tp0.3: 1473395.0000 - val_fp0.3: 266651.0000 - val_tn0.3: 7981144.0000 - val_fn0.3: 109210.0000 - val_precision0.3: 0.8468 - val_recall0.3: 0.9310 - val_tp0.5: 1410224.0000 - val_fp
    0.5: 150868.0000 - val_tn0.5: 8096927.0000 - val_fn0.5: 172381.0000 - val_precision0.5: 0.9034 - val_recall0.5: 0.8911 - val_tp0.7: 1334322.0000 - val_fp0.7: 77751.0000 - val_tn0.7: 817004
    4.0000 - val_fn0.7: 248283.0000 - val_precision0.7: 0.9449 - val_recall0.7: 0.8431 - val_tp0.9: 1186212.0000 - val_fp0.9: 22796.0000 - val_tn0.9: 8224999.0000 - val_fn0.9: 396393.0000 - va
    l_precision0.9: 0.9811 - val_recall0.9: 0.7495 - val_accuracy: 0.9671 - val_auc: 0.9782 - val_f1: 0.2773
    --- Running training session 57/140
    {'hp_epochs': 20, 'hp_batch_size': 16, 'hp_scaler': 'standard', 'hp_n_levels': 3, 'hp_first_filters': 64, 'hp_pool_size': 2, 'hp_input_size': 16384, 'hp_lr_start': 0.05209134418613575, 'hp
    _lr_power': 5.0}
    --- repeat #: 1
    input - shape:   (None, 16384, 1)
    output - shape:  (None, 16384, 1)
    Epoch 1/20
    300/300 [==============================] - 131s 378ms/step - loss: 1.1698 - tp0.1: 9792293.0000 - fp0.1: 16380066.0000 - tn0.1: 50293360.0000 - fn0.1: 2177474.0000 - precision0.1: 0.3741 -
     recall0.1: 0.8181 - tp0.3: 8106203.0000 - fp0.3: 5386658.0000 - tn0.3: 61286792.0000 - fn0.3: 3863564.0000 - precision0.3: 0.6008 - recall0.3: 0.6772 - tp0.5: 6816180.0000 - fp0.5: 244709
    8.0000 - tn0.5: 64226336.0000 - fn0.5: 5153587.0000 - precision0.5: 0.7358 - recall0.5: 0.5694 - tp0.7: 5263785.0000 - fp0.7: 811449.0000 - tn0.7: 65861992.0000 - fn0.7: 6705982.0000 - pre
    cision0.7: 0.8664 - recall0.7: 0.4398 - tp0.9: 3484091.0000 - fp0.9: 160303.0000 - tn0.9: 66513160.0000 - fn0.9: 8485676.0000 - precision0.9: 0.9560 - recall0.9: 0.2911 - accuracy: 0.9034
    - auc: 0.8642 - f1: 0.2642 - val_loss: 2.1107 - val_tp0.1: 3108339.0000 - val_fp0.1: 15105390.0000 - val_tn0.1: 1392556.0000 - val_fn0.1: 54515.0000 - val_precision0.1: 0.1707 - val_recall
    0.1: 0.9828 - val_tp0.3: 2913184.0000 - val_fp0.3: 8671661.0000 - val_tn0.3: 7826285.0000 - val_fn0.3: 249670.0000 - val_precision0.3: 0.2515 - val_recall0.3: 0.9211 - val_tp0.5: 2807049.0
    000 - val_fp0.5: 6353778.0000 - val_tn0.5: 10144168.0000 - val_fn0.5: 355805.0000 - val_precision0.5: 0.3064 - val_recall0.5: 0.8875 - val_tp0.7: 2699435.0000 - val_fp0.7: 4581266.0000 - v
    al_tn0.7: 11916680.0000 - val_fn0.7: 463419.0000 - val_precision0.7: 0.3708 - val_recall0.7: 0.8535 - val_tp0.9: 2536045.0000 - val_fp0.9: 2954857.0000 - val_tn0.9: 13543089.0000 - val_fn0
    .9: 626809.0000 - val_precision0.9: 0.4619 - val_recall0.9: 0.8018 - val_accuracy: 0.6587 - val_auc: 0.8551 - val_f1: 0.2772
    Epoch 2/20
    300/300 [==============================] - 110s 367ms/step - loss: 1.1426 - tp0.1: 9916409.0000 - fp0.1: 13548468.0000 - tn0.1: 53124976.0000 - fn0.1: 2053358.0000 - precision0.1: 0.4226 -
     recall0.1: 0.8285 - tp0.3: 8478833.0000 - fp0.3: 4638495.0000 - tn0.3: 62034944.0000 - fn0.3: 3490934.0000 - precision0.3: 0.6464 - recall0.3: 0.7084 - tp0.5: 7332170.0000 - fp0.5: 218002
    8.0000 - tn0.5: 64493416.0000 - fn0.5: 4637597.0000 - precision0.5: 0.7708 - recall0.5: 0.6126 - tp0.7: 5950349.0000 - fp0.7: 846977.0000 - tn0.7: 65826468.0000 - fn0.7: 6019418.0000 - pre
    cision0.7: 0.8754 - recall0.7: 0.4971 - tp0.9: 4092626.0000 - fp0.9: 183109.0000 - tn0.9: 66490320.0000 - fn0.9: 7877141.0000 - precision0.9: 0.9572 - recall0.9: 0.3419 - accuracy: 0.9133
    - auc: 0.8817 - f1: 0.2642 - val_loss: 1.3552 - val_tp0.1: 3018867.0000 - val_fp0.1: 9642342.0000 - val_tn0.1: 6855604.0000 - val_fn0.1: 143987.0000 - val_precision0.1: 0.2384 - val_recall
    0.1: 0.9545 - val_tp0.3: 2882930.0000 - val_fp0.3: 5948958.0000 - val_tn0.3: 10548988.0000 - val_fn0.3: 279924.0000 - val_precision0.3: 0.3264 - val_recall0.3: 0.9115 - val_tp0.5: 2710540.
    0000 - val_fp0.5: 3482144.0000 - val_tn0.5: 13015802.0000 - val_fn0.5: 452314.0000 - val_precision0.5: 0.4377 - val_recall0.5: 0.8570 - val_tp0.7: 2470093.0000 - val_fp0.7: 1883006.0000 -
    val_tn0.7: 14614940.0000 - val_fn0.7: 692761.0000 - val_precision0.7: 0.5674 - val_recall0.7: 0.7810 - val_tp0.9: 2057360.0000 - val_fp0.9: 785358.0000 - val_tn0.9: 15712588.0000 - val_fn0
    .9: 1105494.0000 - val_precision0.9: 0.7237 - val_recall0.9: 0.6505 - val_accuracy: 0.7999 - val_auc: 0.8997 - val_f1: 0.2772
    Epoch 3/20
    300/300 [==============================] - 110s 367ms/step - loss: 1.1129 - tp0.1: 10053816.0000 - fp0.1: 11055564.0000 - tn0.1: 55617880.0000 - fn0.1: 1915951.0000 - precision0.1: 0.4763
    - recall0.1: 0.8399 - tp0.3: 8934470.0000 - fp0.3: 4064684.0000 - tn0.3: 62608744.0000 - fn0.3: 3035297.0000 - precision0.3: 0.6873 - recall0.3: 0.7464 - tp0.5: 8010627.0000 - fp0.5: 21114
    72.0000 - tn0.5: 64561976.0000 - fn0.5: 3959140.0000 - precision0.5: 0.7914 - recall0.5: 0.6692 - tp0.7: 6744537.0000 - fp0.7: 920340.0000 - tn0.7: 65753096.0000 - fn0.7: 5225230.0000 - pr
    ecision0.7: 0.8799 - recall0.7: 0.5635 - tp0.9: 4739266.0000 - fp0.9: 210718.0000 - tn0.9: 66462700.0000 - fn0.9: 7230501.0000 - precision0.9: 0.9574 - recall0.9: 0.3959 - accuracy: 0.9228
     - auc: 0.9043 - f1: 0.2642 - val_loss: 1.1474 - val_tp0.1: 2967564.0000 - val_fp0.1: 6983406.0000 - val_tn0.1: 9514540.0000 - val_fn0.1: 195290.0000 - val_precision0.1: 0.2982 - val_recal
    l0.1: 0.9383 - val_tp0.3: 2788083.0000 - val_fp0.3: 2981666.0000 - val_tn0.3: 13516280.0000 - val_fn0.3: 374771.0000 - val_precision0.3: 0.4832 - val_recall0.3: 0.8815 - val_tp0.5: 2559864
    .0000 - val_fp0.5: 1470225.0000 - val_tn0.5: 15027721.0000 - val_fn0.5: 602990.0000 - val_precision0.5: 0.6352 - val_recall0.5: 0.8094 - val_tp0.7: 2204980.0000 - val_fp0.7: 672610.0000 -
    val_tn0.7: 15825336.0000 - val_fn0.7: 957874.0000 - val_precision0.7: 0.7663 - val_recall0.7: 0.6971 - val_tp0.9: 1441024.0000 - val_fp0.9: 161378.0000 - val_tn0.9: 16336568.0000 - val_fn0
    .9: 1721830.0000 - val_precision0.9: 0.8993 - val_recall0.9: 0.4556 - val_accuracy: 0.8946 - val_auc: 0.9228 - val_f1: 0.2772
    Epoch 4/20
    300/300 [==============================] - 110s 367ms/step - loss: 0.9937 - tp0.1: 10107575.0000 - fp0.1: 10632681.0000 - tn0.1: 56040784.0000 - fn0.1: 1862192.0000 - precision0.1: 0.4873
    - recall0.1: 0.8444 - tp0.3: 9018636.0000 - fp0.3: 3861193.0000 - tn0.3: 62812244.0000 - fn0.3: 2951131.0000 - precision0.3: 0.7002 - recall0.3: 0.7535 - tp0.5: 8145541.0000 - fp0.5: 20363
    45.0000 - tn0.5: 64637076.0000 - fn0.5: 3824226.0000 - precision0.5: 0.8000 - recall0.5: 0.6805 - tp0.7: 6999441.0000 - fp0.7: 948132.0000 - tn0.7: 65725288.0000 - fn0.7: 4970326.0000 - pr
    ecision0.7: 0.8807 - recall0.7: 0.5848 - tp0.9: 4926381.0000 - fp0.9: 215719.0000 - tn0.9: 66457736.0000 - fn0.9: 7043386.0000 - precision0.9: 0.9580 - recall0.9: 0.4116 - accuracy: 0.9255
     - auc: 0.9063 - f1: 0.2642 - val_loss: 0.8383 - val_tp0.1: 2770907.0000 - val_fp0.1: 3233641.0000 - val_tn0.1: 13264305.0000 - val_fn0.1: 391947.0000 - val_precision0.1: 0.4615 - val_reca
    ll0.1: 0.8761 - val_tp0.3: 2695153.0000 - val_fp0.3: 2457686.0000 - val_tn0.3: 14040260.0000 - val_fn0.3: 467701.0000 - val_precision0.3: 0.5230 - val_recall0.3: 0.8521 - val_tp0.5: 262682
    4.0000 - val_fp0.5: 1996722.0000 - val_tn0.5: 14501224.0000 - val_fn0.5: 536030.0000 - val_precision0.5: 0.5681 - val_recall0.5: 0.8305 - val_tp0.7: 2543169.0000 - val_fp0.7: 1571034.0000
    - val_tn0.7: 14926912.0000 - val_fn0.7: 619685.0000 - val_precision0.7: 0.6181 - val_recall0.7: 0.8041 - val_tp0.9: 2364543.0000 - val_fp0.9: 988221.0000 - val_tn0.9: 15509725.0000 - val_f
    n0.9: 798311.0000 - val_precision0.9: 0.7053 - val_recall0.9: 0.7476 - val_accuracy: 0.8712 - val_auc: 0.9011 - val_f1: 0.2772
    Epoch 5/20
    300/300 [==============================] - 110s 366ms/step - loss: 0.6515 - tp0.1: 10120253.0000 - fp0.1: 9513561.0000 - tn0.1: 57159892.0000 - fn0.1: 1849514.0000 - precision0.1: 0.5155 -
     recall0.1: 0.8455 - tp0.3: 9146172.0000 - fp0.3: 3874592.0000 - tn0.3: 62798872.0000 - fn0.3: 2823595.0000 - precision0.3: 0.7024 - recall0.3: 0.7641 - tp0.5: 8285447.0000 - fp0.5: 203234
    9.0000 - tn0.5: 64641116.0000 - fn0.5: 3684320.0000 - precision0.5: 0.8030 - recall0.5: 0.6922 - tp0.7: 7115867.0000 - fp0.7: 936057.0000 - tn0.7: 65737360.0000 - fn0.7: 4853900.0000 - pre
    cision0.7: 0.8837 - recall0.7: 0.5945 - tp0.9: 5021875.0000 - fp0.9: 210954.0000 - tn0.9: 66462488.0000 - fn0.9: 6947892.0000 - precision0.9: 0.9597 - recall0.9: 0.4195 - accuracy: 0.9273
    - auc: 0.9005 - f1: 0.2642 - val_loss: 0.5790 - val_tp0.1: 2786753.0000 - val_fp0.1: 2338059.0000 - val_tn0.1: 14159887.0000 - val_fn0.1: 376101.0000 - val_precision0.1: 0.5438 - val_recal
    l0.1: 0.8811 - val_tp0.3: 2442329.0000 - val_fp0.3: 767449.0000 - val_tn0.3: 15730497.0000 - val_fn0.3: 720525.0000 - val_precision0.3: 0.7609 - val_recall0.3: 0.7722 - val_tp0.5: 2227274.
    0000 - val_fp0.5: 432048.0000 - val_tn0.5: 16065898.0000 - val_fn0.5: 935580.0000 - val_precision0.5: 0.8375 - val_recall0.5: 0.7042 - val_tp0.7: 1917679.0000 - val_fp0.7: 203533.0000 - va
    l_tn0.7: 16294413.0000 - val_fn0.7: 1245175.0000 - val_precision0.7: 0.9040 - val_recall0.7: 0.6063 - val_tp0.9: 1380335.0000 - val_fp0.9: 47607.0000 - val_tn0.9: 16450339.0000 - val_fn0.9
    : 1782519.0000 - val_precision0.9: 0.9667 - val_recall0.9: 0.4364 - val_accuracy: 0.9304 - val_auc: 0.9216 - val_f1: 0.2772
    Epoch 6/20
    300/300 [==============================] - 110s 367ms/step - loss: 0.6145 - tp0.1: 10221226.0000 - fp0.1: 9054753.0000 - tn0.1: 57618652.0000 - fn0.1: 1748541.0000 - precision0.1: 0.5303 -
     recall0.1: 0.8539 - tp0.3: 9253443.0000 - fp0.3: 3526719.0000 - tn0.3: 63146724.0000 - fn0.3: 2716324.0000 - precision0.3: 0.7240 - recall0.3: 0.7731 - tp0.5: 8544655.0000 - fp0.5: 198031
    7.0000 - tn0.5: 64693120.0000 - fn0.5: 3425112.0000 - precision0.5: 0.8118 - recall0.5: 0.7139 - tp0.7: 7486882.0000 - fp0.7: 931061.0000 - tn0.7: 65742384.0000 - fn0.7: 4482885.0000 - pre
    cision0.7: 0.8894 - recall0.7: 0.6255 - tp0.9: 5438772.0000 - fp0.9: 216030.0000 - tn0.9: 66457408.0000 - fn0.9: 6530995.0000 - precision0.9: 0.9618 - recall0.9: 0.4544 - accuracy: 0.9313
    - auc: 0.9073 - f1: 0.2642 - val_loss: 0.5665 - val_tp0.1: 2863152.0000 - val_fp0.1: 2970240.0000 - val_tn0.1: 13527706.0000 - val_fn0.1: 299702.0000 - val_precision0.1: 0.4908 - val_recal
    l0.1: 0.9052 - val_tp0.3: 2546439.0000 - val_fp0.3: 915799.0000 - val_tn0.3: 15582147.0000 - val_fn0.3: 616415.0000 - val_precision0.3: 0.7355 - val_recall0.3: 0.8051 - val_tp0.5: 2318053.
    0000 - val_fp0.5: 477192.0000 - val_tn0.5: 16020754.0000 - val_fn0.5: 844801.0000 - val_precision0.5: 0.8293 - val_recall0.5: 0.7329 - val_tp0.7: 1952977.0000 - val_fp0.7: 192829.0000 - va
    l_tn0.7: 16305117.0000 - val_fn0.7: 1209877.0000 - val_precision0.7: 0.9101 - val_recall0.7: 0.6175 - val_tp0.9: 1328714.0000 - val_fp0.9: 31660.0000 - val_tn0.9: 16466286.0000 - val_fn0.9
    : 1834140.0000 - val_precision0.9: 0.9767 - val_recall0.9: 0.4201 - val_accuracy: 0.9328 - val_auc: 0.9303 - val_f1: 0.2772
    Epoch 7/20
    300/300 [==============================] - 110s 367ms/step - loss: 0.6010 - tp0.1: 10240926.0000 - fp0.1: 8765134.0000 - tn0.1: 57908304.0000 - fn0.1: 1728841.0000 - precision0.1: 0.5388 -
     recall0.1: 0.8556 - tp0.3: 9298657.0000 - fp0.3: 3414297.0000 - tn0.3: 63259112.0000 - fn0.3: 2671110.0000 - precision0.3: 0.7314 - recall0.3: 0.7768 - tp0.5: 8600118.0000 - fp0.5: 190447
    1.0000 - tn0.5: 64768972.0000 - fn0.5: 3369649.0000 - precision0.5: 0.8187 - recall0.5: 0.7185 - tp0.7: 7578223.0000 - fp0.7: 896266.0000 - tn0.7: 65777152.0000 - fn0.7: 4391544.0000 - pre
    cision0.7: 0.8942 - recall0.7: 0.6331 - tp0.9: 5575318.0000 - fp0.9: 206423.0000 - tn0.9: 66467012.0000 - fn0.9: 6394449.0000 - precision0.9: 0.9643 - recall0.9: 0.4658 - accuracy: 0.9329
    - auc: 0.9091 - f1: 0.2642 - val_loss: 0.6136 - val_tp0.1: 2833227.0000 - val_fp0.1: 2966233.0000 - val_tn0.1: 13531713.0000 - val_fn0.1: 329627.0000 - val_precision0.1: 0.4885 - val_recal
    l0.1: 0.8958 - val_tp0.3: 2657099.0000 - val_fp0.3: 1503941.0000 - val_tn0.3: 14994005.0000 - val_fn0.3: 505755.0000 - val_precision0.3: 0.6386 - val_recall0.3: 0.8401 - val_tp0.5: 2521378
    .0000 - val_fp0.5: 979303.0000 - val_tn0.5: 15518643.0000 - val_fn0.5: 641476.0000 - val_precision0.5: 0.7203 - val_recall0.5: 0.7972 - val_tp0.7: 2355434.0000 - val_fp0.7: 617852.0000 - v
    al_tn0.7: 15880094.0000 - val_fn0.7: 807420.0000 - val_precision0.7: 0.7922 - val_recall0.7: 0.7447 - val_tp0.9: 2044932.0000 - val_fp0.9: 285145.0000 - val_tn0.9: 16212801.0000 - val_fn0.
    9: 1117922.0000 - val_precision0.9: 0.8776 - val_recall0.9: 0.6465 - val_accuracy: 0.9176 - val_auc: 0.9226 - val_f1: 0.2772
    Epoch 8/20
    300/300 [==============================] - 110s 367ms/step - loss: 0.5874 - tp0.1: 10313883.0000 - fp0.1: 8946511.0000 - tn0.1: 57726912.0000 - fn0.1: 1655884.0000 - precision0.1: 0.5355 -
     recall0.1: 0.8617 - tp0.3: 9372263.0000 - fp0.3: 3402993.0000 - tn0.3: 63270404.0000 - fn0.3: 2597504.0000 - precision0.3: 0.7336 - recall0.3: 0.7830 - tp0.5: 8691543.0000 - fp0.5: 189512
    5.0000 - tn0.5: 64778312.0000 - fn0.5: 3278224.0000 - precision0.5: 0.8210 - recall0.5: 0.7261 - tp0.7: 7678420.0000 - fp0.7: 891412.0000 - tn0.7: 65782052.0000 - fn0.7: 4291347.0000 - pre
    cision0.7: 0.8960 - recall0.7: 0.6415 - tp0.9: 5663701.0000 - fp0.9: 205730.0000 - tn0.9: 66467716.0000 - fn0.9: 6306066.0000 - precision0.9: 0.9649 - recall0.9: 0.4732 - accuracy: 0.9342
    - auc: 0.9123 - f1: 0.2642 - val_loss: 0.5994 - val_tp0.1: 2655699.0000 - val_fp0.1: 1220112.0000 - val_tn0.1: 15277834.0000 - val_fn0.1: 507155.0000 - val_precision0.1: 0.6852 - val_recal
    l0.1: 0.8397 - val_tp0.3: 2195697.0000 - val_fp0.3: 383330.0000 - val_tn0.3: 16114616.0000 - val_fn0.3: 967157.0000 - val_precision0.3: 0.8514 - val_recall0.3: 0.6942 - val_tp0.5: 1922446.
    0000 - val_fp0.5: 203083.0000 - val_tn0.5: 16294863.0000 - val_fn0.5: 1240408.0000 - val_precision0.5: 0.9045 - val_recall0.5: 0.6078 - val_tp0.7: 1577371.0000 - val_fp0.7: 87819.0000 - va
    l_tn0.7: 16410127.0000 - val_fn0.7: 1585483.0000 - val_precision0.7: 0.9473 - val_recall0.7: 0.4987 - val_tp0.9: 1033785.0000 - val_fp0.9: 15309.0000 - val_tn0.9: 16482637.0000 - val_fn0.9
    : 2129069.0000 - val_precision0.9: 0.9854 - val_recall0.9: 0.3269 - val_accuracy: 0.9266 - val_auc: 0.9095 - val_f1: 0.2772
    Epoch 9/20
    300/300 [==============================] - 110s 366ms/step - loss: 0.5779 - tp0.1: 10317324.0000 - fp0.1: 8680261.0000 - tn0.1: 57993180.0000 - fn0.1: 1652443.0000 - precision0.1: 0.5431 -
     recall0.1: 0.8619 - tp0.3: 9391398.0000 - fp0.3: 3311151.0000 - tn0.3: 63362284.0000 - fn0.3: 2578369.0000 - precision0.3: 0.7393 - recall0.3: 0.7846 - tp0.5: 8705608.0000 - fp0.5: 181412
    3.0000 - tn0.5: 64859340.0000 - fn0.5: 3264159.0000 - precision0.5: 0.8276 - recall0.5: 0.7273 - tp0.7: 7731054.0000 - fp0.7: 858059.0000 - tn0.7: 65815376.0000 - fn0.7: 4238713.0000 - pre
    cision0.7: 0.9001 - recall0.7: 0.6459 - tp0.9: 5763101.0000 - fp0.9: 200933.0000 - tn0.9: 66472512.0000 - fn0.9: 6206666.0000 - precision0.9: 0.9663 - recall0.9: 0.4815 - accuracy: 0.9354
    - auc: 0.9133 - f1: 0.2642 - val_loss: 0.5336 - val_tp0.1: 2783751.0000 - val_fp0.1: 1864654.0000 - val_tn0.1: 14633292.0000 - val_fn0.1: 379103.0000 - val_precision0.1: 0.5989 - val_recal
    l0.1: 0.8801 - val_tp0.3: 2518451.0000 - val_fp0.3: 718668.0000 - val_tn0.3: 15779278.0000 - val_fn0.3: 644403.0000 - val_precision0.3: 0.7780 - val_recall0.3: 0.7963 - val_tp0.5: 2312604.
    0000 - val_fp0.5: 409725.0000 - val_tn0.5: 16088221.0000 - val_fn0.5: 850250.0000 - val_precision0.5: 0.8495 - val_recall0.5: 0.7312 - val_tp0.7: 2020053.0000 - val_fp0.7: 204600.0000 - va
    l_tn0.7: 16293346.0000 - val_fn0.7: 1142801.0000 - val_precision0.7: 0.9080 - val_recall0.7: 0.6387 - val_tp0.9: 1443951.0000 - val_fp0.9: 46830.0000 - val_tn0.9: 16451116.0000 - val_fn0.9
    : 1718903.0000 - val_precision0.9: 0.9686 - val_recall0.9: 0.4565 - val_accuracy: 0.9359 - val_auc: 0.9250 - val_f1: 0.2772
    Epoch 10/20
    300/300 [==============================] - 110s 367ms/step - loss: 0.5662 - tp0.1: 10352395.0000 - fp0.1: 8545400.0000 - tn0.1: 58128032.0000 - fn0.1: 1617372.0000 - precision0.1: 0.5478 -
     recall0.1: 0.8649 - tp0.3: 9453462.0000 - fp0.3: 3297727.0000 - tn0.3: 63375704.0000 - fn0.3: 2516305.0000 - precision0.3: 0.7414 - recall0.3: 0.7898 - tp0.5: 8780933.0000 - fp0.5: 180115
    0.0000 - tn0.5: 64872296.0000 - fn0.5: 3188834.0000 - precision0.5: 0.8298 - recall0.5: 0.7336 - tp0.7: 7844065.0000 - fp0.7: 860857.0000 - tn0.7: 65812548.0000 - fn0.7: 4125702.0000 - pre
    cision0.7: 0.9011 - recall0.7: 0.6553 - tp0.9: 5892491.0000 - fp0.9: 202051.0000 - tn0.9: 66471364.0000 - fn0.9: 6077276.0000 - precision0.9: 0.9668 - recall0.9: 0.4923 - accuracy: 0.9365
    - auc: 0.9156 - f1: 0.2642 - val_loss: 0.5278 - val_tp0.1: 2876358.0000 - val_fp0.1: 2715037.0000 - val_tn0.1: 13782909.0000 - val_fn0.1: 286496.0000 - val_precision0.1: 0.5144 - val_recal
    l0.1: 0.9094 - val_tp0.3: 2636089.0000 - val_fp0.3: 1014420.0000 - val_tn0.3: 15483526.0000 - val_fn0.3: 526765.0000 - val_precision0.3: 0.7221 - val_recall0.3: 0.8335 - val_tp0.5: 2435712
    .0000 - val_fp0.5: 548278.0000 - val_tn0.5: 15949668.0000 - val_fn0.5: 727142.0000 - val_precision0.5: 0.8163 - val_recall0.5: 0.7701 - val_tp0.7: 2160433.0000 - val_fp0.7: 261165.0000 - v
    al_tn0.7: 16236781.0000 - val_fn0.7: 1002421.0000 - val_precision0.7: 0.8922 - val_recall0.7: 0.6831 - val_tp0.9: 1556420.0000 - val_fp0.9: 50492.0000 - val_tn0.9: 16447454.0000 - val_fn0.
    9: 1606434.0000 - val_precision0.9: 0.9686 - val_recall0.9: 0.4921 - val_accuracy: 0.9351 - val_auc: 0.9348 - val_f1: 0.2772
    Epoch 11/20
    300/300 [==============================] - 110s 367ms/step - loss: 0.5610 - tp0.1: 10373608.0000 - fp0.1: 8572388.0000 - tn0.1: 58101040.0000 - fn0.1: 1596159.0000 - precision0.1: 0.5475 -
     recall0.1: 0.8667 - tp0.3: 9468020.0000 - fp0.3: 3221384.0000 - tn0.3: 63452048.0000 - fn0.3: 2501747.0000 - precision0.3: 0.7461 - recall0.3: 0.7910 - tp0.5: 8789688.0000 - fp0.5: 173793
    5.0000 - tn0.5: 64935508.0000 - fn0.5: 3180079.0000 - precision0.5: 0.8349 - recall0.5: 0.7343 - tp0.7: 7882409.0000 - fp0.7: 841710.0000 - tn0.7: 65831744.0000 - fn0.7: 4087358.0000 - pre
    cision0.7: 0.9035 - recall0.7: 0.6585 - tp0.9: 5966113.0000 - fp0.9: 196483.0000 - tn0.9: 66476952.0000 - fn0.9: 6003654.0000 - precision0.9: 0.9681 - recall0.9: 0.4984 - accuracy: 0.9375
    - auc: 0.9166 - f1: 0.2642 - val_loss: 0.5227 - val_tp0.1: 2833580.0000 - val_fp0.1: 2220007.0000 - val_tn0.1: 14277939.0000 - val_fn0.1: 329274.0000 - val_precision0.1: 0.5607 - val_recal
    l0.1: 0.8959 - val_tp0.3: 2545954.0000 - val_fp0.3: 744480.0000 - val_tn0.3: 15753466.0000 - val_fn0.3: 616900.0000 - val_precision0.3: 0.7737 - val_recall0.3: 0.8050 - val_tp0.5: 2303452.
    0000 - val_fp0.5: 376141.0000 - val_tn0.5: 16121805.0000 - val_fn0.5: 859402.0000 - val_precision0.5: 0.8596 - val_recall0.5: 0.7283 - val_tp0.7: 1971041.0000 - val_fp0.7: 160456.0000 - va
    l_tn0.7: 16337490.0000 - val_fn0.7: 1191813.0000 - val_precision0.7: 0.9247 - val_recall0.7: 0.6232 - val_tp0.9: 1375228.0000 - val_fp0.9: 28055.0000 - val_tn0.9: 16469891.0000 - val_fn0.9
    : 1787626.0000 - val_precision0.9: 0.9800 - val_recall0.9: 0.4348 - val_accuracy: 0.9372 - val_auc: 0.9312 - val_f1: 0.2772
    Epoch 12/20
    300/300 [==============================] - 110s 366ms/step - loss: 0.5516 - tp0.1: 10421559.0000 - fp0.1: 8658554.0000 - tn0.1: 58014880.0000 - fn0.1: 1548208.0000 - precision0.1: 0.5462 -
     recall0.1: 0.8707 - tp0.3: 9509353.0000 - fp0.3: 3240736.0000 - tn0.3: 63432728.0000 - fn0.3: 2460414.0000 - precision0.3: 0.7458 - recall0.3: 0.7944 - tp0.5: 8835728.0000 - fp0.5: 174437
    1.0000 - tn0.5: 64929060.0000 - fn0.5: 3134039.0000 - precision0.5: 0.8351 - recall0.5: 0.7382 - tp0.7: 7935785.0000 - fp0.7: 836444.0000 - tn0.7: 65836992.0000 - fn0.7: 4033982.0000 - pre
    cision0.7: 0.9046 - recall0.7: 0.6630 - tp0.9: 6021972.0000 - fp0.9: 194282.0000 - tn0.9: 66479164.0000 - fn0.9: 5947795.0000 - precision0.9: 0.9687 - recall0.9: 0.5031 - accuracy: 0.9380
    - auc: 0.9190 - f1: 0.2642 - val_loss: 0.5259 - val_tp0.1: 2813028.0000 - val_fp0.1: 2078752.0000 - val_tn0.1: 14419194.0000 - val_fn0.1: 349826.0000 - val_precision0.1: 0.5751 - val_recal
    l0.1: 0.8894 - val_tp0.3: 2528314.0000 - val_fp0.3: 706366.0000 - val_tn0.3: 15791580.0000 - val_fn0.3: 634540.0000 - val_precision0.3: 0.7816 - val_recall0.3: 0.7994 - val_tp0.5: 2316024.
    0000 - val_fp0.5: 388662.0000 - val_tn0.5: 16109284.0000 - val_fn0.5: 846830.0000 - val_precision0.5: 0.8563 - val_recall0.5: 0.7323 - val_tp0.7: 2044271.0000 - val_fp0.7: 193248.0000 - va
    l_tn0.7: 16304698.0000 - val_fn0.7: 1118583.0000 - val_precision0.7: 0.9136 - val_recall0.7: 0.6463 - val_tp0.9: 1538039.0000 - val_fp0.9: 48557.0000 - val_tn0.9: 16449389.0000 - val_fn0.9
    : 1624815.0000 - val_precision0.9: 0.9694 - val_recall0.9: 0.4863 - val_accuracy: 0.9372 - val_auc: 0.9290 - val_f1: 0.2772
    Epoch 13/20
    300/300 [==============================] - 110s 366ms/step - loss: 0.5454 - tp0.1: 10439462.0000 - fp0.1: 8578643.0000 - tn0.1: 58094796.0000 - fn0.1: 1530305.0000 - precision0.1: 0.5489 -
     recall0.1: 0.8722 - tp0.3: 9511162.0000 - fp0.3: 3179906.0000 - tn0.3: 63493568.0000 - fn0.3: 2458605.0000 - precision0.3: 0.7494 - recall0.3: 0.7946 - tp0.5: 8819389.0000 - fp0.5: 168064
    2.0000 - tn0.5: 64992792.0000 - fn0.5: 3150378.0000 - precision0.5: 0.8399 - recall0.5: 0.7368 - tp0.7: 7930729.0000 - fp0.7: 810316.0000 - tn0.7: 65863144.0000 - fn0.7: 4039038.0000 - pre
    cision0.7: 0.9073 - recall0.7: 0.6626 - tp0.9: 6037837.0000 - fp0.9: 190230.0000 - tn0.9: 66483196.0000 - fn0.9: 5931930.0000 - precision0.9: 0.9695 - recall0.9: 0.5044 - accuracy: 0.9386
    - auc: 0.9201 - f1: 0.2642 - val_loss: 0.5152 - val_tp0.1: 2825402.0000 - val_fp0.1: 2086657.0000 - val_tn0.1: 14411289.0000 - val_fn0.1: 337452.0000 - val_precision0.1: 0.5752 - val_recal
    l0.1: 0.8933 - val_tp0.3: 2573708.0000 - val_fp0.3: 777952.0000 - val_tn0.3: 15719994.0000 - val_fn0.3: 589146.0000 - val_precision0.3: 0.7679 - val_recall0.3: 0.8137 - val_tp0.5: 2360881.
    0000 - val_fp0.5: 416427.0000 - val_tn0.5: 16081519.0000 - val_fn0.5: 801973.0000 - val_precision0.5: 0.8501 - val_recall0.5: 0.7464 - val_tp0.7: 2074036.0000 - val_fp0.7: 196402.0000 - va
    l_tn0.7: 16301544.0000 - val_fn0.7: 1088818.0000 - val_precision0.7: 0.9135 - val_recall0.7: 0.6557 - val_tp0.9: 1511886.0000 - val_fp0.9: 41512.0000 - val_tn0.9: 16456434.0000 - val_fn0.9
    : 1650968.0000 - val_precision0.9: 0.9733 - val_recall0.9: 0.4780 - val_accuracy: 0.9380 - val_auc: 0.9310 - val_f1: 0.2772
    Epoch 14/20
    300/300 [==============================] - 110s 366ms/step - loss: 0.5443 - tp0.1: 10444511.0000 - fp0.1: 8600961.0000 - tn0.1: 58072476.0000 - fn0.1: 1525256.0000 - precision0.1: 0.5484 -
     recall0.1: 0.8726 - tp0.3: 9538913.0000 - fp0.3: 3252925.0000 - tn0.3: 63420496.0000 - fn0.3: 2430854.0000 - precision0.3: 0.7457 - recall0.3: 0.7969 - tp0.5: 8849106.0000 - fp0.5: 171047
    5.0000 - tn0.5: 64962976.0000 - fn0.5: 3120661.0000 - precision0.5: 0.8380 - recall0.5: 0.7393 - tp0.7: 7973420.0000 - fp0.7: 827235.0000 - tn0.7: 65846200.0000 - fn0.7: 3996347.0000 - pre
    cision0.7: 0.9060 - recall0.7: 0.6661 - tp0.9: 6091812.0000 - fp0.9: 195928.0000 - tn0.9: 66477488.0000 - fn0.9: 5877955.0000 - precision0.9: 0.9688 - recall0.9: 0.5089 - accuracy: 0.9386
    - auc: 0.9203 - f1: 0.2642 - val_loss: 0.5191 - val_tp0.1: 2826998.0000 - val_fp0.1: 2155571.0000 - val_tn0.1: 14342375.0000 - val_fn0.1: 335856.0000 - val_precision0.1: 0.5674 - val_recal
    l0.1: 0.8938 - val_tp0.3: 2586837.0000 - val_fp0.3: 829620.0000 - val_tn0.3: 15668326.0000 - val_fn0.3: 576017.0000 - val_precision0.3: 0.7572 - val_recall0.3: 0.8179 - val_tp0.5: 2383398.
    0000 - val_fp0.5: 453397.0000 - val_tn0.5: 16044549.0000 - val_fn0.5: 779456.0000 - val_precision0.5: 0.8402 - val_recall0.5: 0.7536 - val_tp0.7: 2122357.0000 - val_fp0.7: 227187.0000 - va
    l_tn0.7: 16270759.0000 - val_fn0.7: 1040497.0000 - val_precision0.7: 0.9033 - val_recall0.7: 0.6710 - val_tp0.9: 1608918.0000 - val_fp0.9: 56008.0000 - val_tn0.9: 16441938.0000 - val_fn0.9
    : 1553936.0000 - val_precision0.9: 0.9664 - val_recall0.9: 0.5087 - val_accuracy: 0.9373 - val_auc: 0.9305 - val_f1: 0.2772
    Epoch 15/20
    300/300 [==============================] - 110s 367ms/step - loss: 0.5447 - tp0.1: 10433601.0000 - fp0.1: 8522557.0000 - tn0.1: 58150872.0000 - fn0.1: 1536166.0000 - precision0.1: 0.5504 -
     recall0.1: 0.8717 - tp0.3: 9526040.0000 - fp0.3: 3191360.0000 - tn0.3: 63482064.0000 - fn0.3: 2443727.0000 - precision0.3: 0.7491 - recall0.3: 0.7958 - tp0.5: 8837031.0000 - fp0.5: 167970
    9.0000 - tn0.5: 64993724.0000 - fn0.5: 3132736.0000 - precision0.5: 0.8403 - recall0.5: 0.7383 - tp0.7: 7952386.0000 - fp0.7: 812351.0000 - tn0.7: 65861096.0000 - fn0.7: 4017381.0000 - pre
    cision0.7: 0.9073 - recall0.7: 0.6644 - tp0.9: 6066606.0000 - fp0.9: 190872.0000 - tn0.9: 66482528.0000 - fn0.9: 5903161.0000 - precision0.9: 0.9695 - recall0.9: 0.5068 - accuracy: 0.9388
    - auc: 0.9200 - f1: 0.2642 - val_loss: 0.5173 - val_tp0.1: 2837321.0000 - val_fp0.1: 2234345.0000 - val_tn0.1: 14263601.0000 - val_fn0.1: 325533.0000 - val_precision0.1: 0.5594 - val_recal
    l0.1: 0.8971 - val_tp0.3: 2592118.0000 - val_fp0.3: 832258.0000 - val_tn0.3: 15665688.0000 - val_fn0.3: 570736.0000 - val_precision0.3: 0.7570 - val_recall0.3: 0.8196 - val_tp0.5: 2384808.
    0000 - val_fp0.5: 447800.0000 - val_tn0.5: 16050146.0000 - val_fn0.5: 778046.0000 - val_precision0.5: 0.8419 - val_recall0.5: 0.7540 - val_tp0.7: 2112938.0000 - val_fp0.7: 217396.0000 - va
    l_tn0.7: 16280550.0000 - val_fn0.7: 1049916.0000 - val_precision0.7: 0.9067 - val_recall0.7: 0.6680 - val_tp0.9: 1562455.0000 - val_fp0.9: 48138.0000 - val_tn0.9: 16449808.0000 - val_fn0.9
    : 1600399.0000 - val_precision0.9: 0.9701 - val_recall0.9: 0.4940 - val_accuracy: 0.9377 - val_auc: 0.9318 - val_f1: 0.2772
    Epoch 16/20
    300/300 [==============================] - 110s 367ms/step - loss: 0.5470 - tp0.1: 10446770.0000 - fp0.1: 8763433.0000 - tn0.1: 57909992.0000 - fn0.1: 1522997.0000 - precision0.1: 0.5438 -
     recall0.1: 0.8728 - tp0.3: 9531541.0000 - fp0.3: 3286354.0000 - tn0.3: 63387068.0000 - fn0.3: 2438226.0000 - precision0.3: 0.7436 - recall0.3: 0.7963 - tp0.5: 8849773.0000 - fp0.5: 174540
    6.0000 - tn0.5: 64928032.0000 - fn0.5: 3119994.0000 - precision0.5: 0.8353 - recall0.5: 0.7393 - tp0.7: 7984098.0000 - fp0.7: 851547.0000 - tn0.7: 65821888.0000 - fn0.7: 3985669.0000 - pre
    cision0.7: 0.9036 - recall0.7: 0.6670 - tp0.9: 6119565.0000 - fp0.9: 198791.0000 - tn0.9: 66474608.0000 - fn0.9: 5850202.0000 - precision0.9: 0.9685 - recall0.9: 0.5113 - accuracy: 0.9381
    - auc: 0.9200 - f1: 0.2642 - val_loss: 0.5199 - val_tp0.1: 2830415.0000 - val_fp0.1: 2198213.0000 - val_tn0.1: 14299733.0000 - val_fn0.1: 332439.0000 - val_precision0.1: 0.5629 - val_recal
    l0.1: 0.8949 - val_tp0.3: 2585794.0000 - val_fp0.3: 826003.0000 - val_tn0.3: 15671943.0000 - val_fn0.3: 577060.0000 - val_precision0.3: 0.7579 - val_recall0.3: 0.8176 - val_tp0.5: 2380750.
    0000 - val_fp0.5: 448412.0000 - val_tn0.5: 16049534.0000 - val_fn0.5: 782104.0000 - val_precision0.5: 0.8415 - val_recall0.5: 0.7527 - val_tp0.7: 2117268.0000 - val_fp0.7: 222683.0000 - va
    l_tn0.7: 16275263.0000 - val_fn0.7: 1045586.0000 - val_precision0.7: 0.9048 - val_recall0.7: 0.6694 - val_tp0.9: 1600774.0000 - val_fp0.9: 54219.0000 - val_tn0.9: 16443727.0000 - val_fn0.9
    : 1562080.0000 - val_precision0.9: 0.9672 - val_recall0.9: 0.5061 - val_accuracy: 0.9374 - val_auc: 0.9309 - val_f1: 0.2772
    Epoch 17/20
    300/300 [==============================] - 110s 366ms/step - loss: 0.5479 - tp0.1: 10435195.0000 - fp0.1: 8675069.0000 - tn0.1: 57998372.0000 - fn0.1: 1534572.0000 - precision0.1: 0.5461 -
     recall0.1: 0.8718 - tp0.3: 9510450.0000 - fp0.3: 3192019.0000 - tn0.3: 63481376.0000 - fn0.3: 2459317.0000 - precision0.3: 0.7487 - recall0.3: 0.7945 - tp0.5: 8815376.0000 - fp0.5: 168445
    9.0000 - tn0.5: 64988968.0000 - fn0.5: 3154391.0000 - precision0.5: 0.8396 - recall0.5: 0.7365 - tp0.7: 7941450.0000 - fp0.7: 818142.0000 - tn0.7: 65855296.0000 - fn0.7: 4028317.0000 - pre
    cision0.7: 0.9066 - recall0.7: 0.6635 - tp0.9: 6071147.0000 - fp0.9: 191455.0000 - tn0.9: 66481988.0000 - fn0.9: 5898620.0000 - precision0.9: 0.9694 - recall0.9: 0.5072 - accuracy: 0.9385
    - auc: 0.9197 - f1: 0.2642 - val_loss: 0.5189 - val_tp0.1: 2828825.0000 - val_fp0.1: 2176033.0000 - val_tn0.1: 14321913.0000 - val_fn0.1: 334029.0000 - val_precision0.1: 0.5652 - val_recal
    l0.1: 0.8944 - val_tp0.3: 2585975.0000 - val_fp0.3: 823670.0000 - val_tn0.3: 15674276.0000 - val_fn0.3: 576879.0000 - val_precision0.3: 0.7584 - val_recall0.3: 0.8176 - val_tp0.5: 2383653.
    0000 - val_fp0.5: 450199.0000 - val_tn0.5: 16047747.0000 - val_fn0.5: 779201.0000 - val_precision0.5: 0.8411 - val_recall0.5: 0.7536 - val_tp0.7: 2121977.0000 - val_fp0.7: 224378.0000 - va
    l_tn0.7: 16273568.0000 - val_fn0.7: 1040877.0000 - val_precision0.7: 0.9044 - val_recall0.7: 0.6709 - val_tp0.9: 1604835.0000 - val_fp0.9: 54571.0000 - val_tn0.9: 16443375.0000 - val_fn0.9
    : 1558019.0000 - val_precision0.9: 0.9671 - val_recall0.9: 0.5074 - val_accuracy: 0.9375 - val_auc: 0.9309 - val_f1: 0.2772
    Epoch 18/20
    300/300 [==============================] - 110s 367ms/step - loss: 0.5454 - tp0.1: 10442547.0000 - fp0.1: 8683315.0000 - tn0.1: 57990116.0000 - fn0.1: 1527220.0000 - precision0.1: 0.5460 -
     recall0.1: 0.8724 - tp0.3: 9524521.0000 - fp0.3: 3195203.0000 - tn0.3: 63478232.0000 - fn0.3: 2445246.0000 - precision0.3: 0.7488 - recall0.3: 0.7957 - tp0.5: 8838166.0000 - fp0.5: 167403
    0.0000 - tn0.5: 64999392.0000 - fn0.5: 3131601.0000 - precision0.5: 0.8408 - recall0.5: 0.7384 - tp0.7: 7963121.0000 - fp0.7: 806164.0000 - tn0.7: 65867296.0000 - fn0.7: 4006646.0000 - pre
    cision0.7: 0.9081 - recall0.7: 0.6653 - tp0.9: 6093558.0000 - fp0.9: 189519.0000 - tn0.9: 66483896.0000 - fn0.9: 5876209.0000 - precision0.9: 0.9698 - recall0.9: 0.5091 - accuracy: 0.9389
    - auc: 0.9202 - f1: 0.2642 - val_loss: 0.5184 - val_tp0.1: 2832372.0000 - val_fp0.1: 2204134.0000 - val_tn0.1: 14293812.0000 - val_fn0.1: 330482.0000 - val_precision0.1: 0.5624 - val_recal
    l0.1: 0.8955 - val_tp0.3: 2594182.0000 - val_fp0.3: 844891.0000 - val_tn0.3: 15653055.0000 - val_fn0.3: 568672.0000 - val_precision0.3: 0.7543 - val_recall0.3: 0.8202 - val_tp0.5: 2395380.
    0000 - val_fp0.5: 464502.0000 - val_tn0.5: 16033444.0000 - val_fn0.5: 767474.0000 - val_precision0.5: 0.8376 - val_recall0.5: 0.7573 - val_tp0.7: 2138310.0000 - val_fp0.7: 234231.0000 - va
    l_tn0.7: 16263715.0000 - val_fn0.7: 1024544.0000 - val_precision0.7: 0.9013 - val_recall0.7: 0.6761 - val_tp0.9: 1623105.0000 - val_fp0.9: 57407.0000 - val_tn0.9: 16440539.0000 - val_fn0.9
    : 1539749.0000 - val_precision0.9: 0.9658 - val_recall0.9: 0.5132 - val_accuracy: 0.9373 - val_auc: 0.9313 - val_f1: 0.2772
    Epoch 19/20
    300/300 [==============================] - 110s 366ms/step - loss: 0.5418 - tp0.1: 10456777.0000 - fp0.1: 8641725.0000 - tn0.1: 58031732.0000 - fn0.1: 1512990.0000 - precision0.1: 0.5475 -
     recall0.1: 0.8736 - tp0.3: 9535623.0000 - fp0.3: 3181102.0000 - tn0.3: 63492344.0000 - fn0.3: 2434144.0000 - precision0.3: 0.7498 - recall0.3: 0.7966 - tp0.5: 8844625.0000 - fp0.5: 167217
    5.0000 - tn0.5: 65001264.0000 - fn0.5: 3125142.0000 - precision0.5: 0.8410 - recall0.5: 0.7389 - tp0.7: 7965635.0000 - fp0.7: 804103.0000 - tn0.7: 65869328.0000 - fn0.7: 4004132.0000 - pre
    cision0.7: 0.9083 - recall0.7: 0.6655 - tp0.9: 6091897.0000 - fp0.9: 189993.0000 - tn0.9: 66483440.0000 - fn0.9: 5877870.0000 - precision0.9: 0.9698 - recall0.9: 0.5089 - accuracy: 0.9390
    - auc: 0.9210 - f1: 0.2642 - val_loss: 0.5190 - val_tp0.1: 2831559.0000 - val_fp0.1: 2201859.0000 - val_tn0.1: 14296087.0000 - val_fn0.1: 331295.0000 - val_precision0.1: 0.5626 - val_recal
    l0.1: 0.8953 - val_tp0.3: 2592759.0000 - val_fp0.3: 841402.0000 - val_tn0.3: 15656544.0000 - val_fn0.3: 570095.0000 - val_precision0.3: 0.7550 - val_recall0.3: 0.8198 - val_tp0.5: 2393128.
    0000 - val_fp0.5: 461852.0000 - val_tn0.5: 16036094.0000 - val_fn0.5: 769726.0000 - val_precision0.5: 0.8382 - val_recall0.5: 0.7566 - val_tp0.7: 2135701.0000 - val_fp0.7: 232684.0000 - va
    l_tn0.7: 16265262.0000 - val_fn0.7: 1027153.0000 - val_precision0.7: 0.9018 - val_recall0.7: 0.6752 - val_tp0.9: 1622963.0000 - val_fp0.9: 57421.0000 - val_tn0.9: 16440525.0000 - val_fn0.9
    : 1539891.0000 - val_precision0.9: 0.9658 - val_recall0.9: 0.5131 - val_accuracy: 0.9374 - val_auc: 0.9311 - val_f1: 0.2772
    Epoch 20/20
    300/300 [==============================] - 110s 366ms/step - loss: 0.5454 - tp0.1: 10445310.0000 - fp0.1: 8664704.0000 - tn0.1: 58008732.0000 - fn0.1: 1524457.0000 - precision0.1: 0.5466 -
     recall0.1: 0.8726 - tp0.3: 9522539.0000 - fp0.3: 3185766.0000 - tn0.3: 63487664.0000 - fn0.3: 2447228.0000 - precision0.3: 0.7493 - recall0.3: 0.7955 - tp0.5: 8829214.0000 - fp0.5: 167729
    6.0000 - tn0.5: 64996140.0000 - fn0.5: 3140553.0000 - precision0.5: 0.8404 - recall0.5: 0.7376 - tp0.7: 7945247.0000 - fp0.7: 812426.0000 - tn0.7: 65861008.0000 - fn0.7: 4024520.0000 - pre
    cision0.7: 0.9072 - recall0.7: 0.6638 - tp0.9: 6065229.0000 - fp0.9: 190972.0000 - tn0.9: 66482444.0000 - fn0.9: 5904538.0000 - precision0.9: 0.9695 - recall0.9: 0.5067 - accuracy: 0.9387
    - auc: 0.9203 - f1: 0.2642 - val_loss: 0.5199 - val_tp0.1: 2827480.0000 - val_fp0.1: 2173312.0000 - val_tn0.1: 14324634.0000 - val_fn0.1: 335374.0000 - val_precision0.1: 0.5654 - val_recal
    l0.1: 0.8940 - val_tp0.3: 2584623.0000 - val_fp0.3: 824555.0000 - val_tn0.3: 15673391.0000 - val_fn0.3: 578231.0000 - val_precision0.3: 0.7581 - val_recall0.3: 0.8172 - val_tp0.5: 2381163.
    0000 - val_fp0.5: 449934.0000 - val_tn0.5: 16048012.0000 - val_fn0.5: 781691.0000 - val_precision0.5: 0.8411 - val_recall0.5: 0.7529 - val_tp0.7: 2119470.0000 - val_fp0.7: 224462.0000 - va
    l_tn0.7: 16273484.0000 - val_fn0.7: 1043384.0000 - val_precision0.7: 0.9042 - val_recall0.7: 0.6701 - val_tp0.9: 1605312.0000 - val_fp0.9: 55121.0000 - val_tn0.9: 16442825.0000 - val_fn0.9
    : 1557542.0000 - val_precision0.9: 0.9668 - val_recall0.9: 0.5076 - val_accuracy: 0.9374 - val_auc: 0.9306 - val_f1: 0.2772
    --- Running training session 58/140
    {'hp_epochs': 20, 'hp_batch_size': 16, 'hp_scaler': 'standard', 'hp_n_levels': 3, 'hp_first_filters': 64, 'hp_pool_size': 2, 'hp_input_size': 16384, 'hp_lr_start': 0.05209134418613575, 'hp
    _lr_power': 5.0}
    --- repeat #: 2
    input - shape:   (None, 16384, 1)
    output - shape:  (None, 16384, 1)
    Epoch 1/20
    300/300 [==============================] - 129s 378ms/step - loss: 1.1658 - tp0.1: 9830604.0000 - fp0.1: 16232549.0000 - tn0.1: 50440888.0000 - fn0.1: 2139163.0000 - precision0.1: 0.3772 -
     recall0.1: 0.8213 - tp0.3: 8228708.0000 - fp0.3: 5302541.0000 - tn0.3: 61370904.0000 - fn0.3: 3741059.0000 - precision0.3: 0.6081 - recall0.3: 0.6875 - tp0.5: 6775084.0000 - fp0.5: 211648
    8.0000 - tn0.5: 64556952.0000 - fn0.5: 5194683.0000 - precision0.5: 0.7620 - recall0.5: 0.5660 - tp0.7: 5350792.0000 - fp0.7: 805898.0000 - tn0.7: 65867540.0000 - fn0.7: 6618975.0000 - pre
    cision0.7: 0.8691 - recall0.7: 0.4470 - tp0.9: 3697858.0000 - fp0.9: 194591.0000 - tn0.9: 66478832.0000 - fn0.9: 8271909.0000 - precision0.9: 0.9500 - recall0.9: 0.3089 - accuracy: 0.9070
    - auc: 0.8672 - f1: 0.2642 - val_loss: 1.3447 - val_tp0.1: 3029789.0000 - val_fp0.1: 10720478.0000 - val_tn0.1: 5777468.0000 - val_fn0.1: 133065.0000 - val_precision0.1: 0.2203 - val_recal
    l0.1: 0.9579 - val_tp0.3: 2725924.0000 - val_fp0.3: 5062932.0000 - val_tn0.3: 11435014.0000 - val_fn0.3: 436930.0000 - val_precision0.3: 0.3500 - val_recall0.3: 0.8619 - val_tp0.5: 2519324
    .0000 - val_fp0.5: 3056247.0000 - val_tn0.5: 13441699.0000 - val_fn0.5: 643530.0000 - val_precision0.5: 0.4519 - val_recall0.5: 0.7965 - val_tp0.7: 2295185.0000 - val_fp0.7: 1804933.0000 -
     val_tn0.7: 14693013.0000 - val_fn0.7: 867669.0000 - val_precision0.7: 0.5598 - val_recall0.7: 0.7257 - val_tp0.9: 1906967.0000 - val_fp0.9: 746282.0000 - val_tn0.9: 15751664.0000 - val_fn
    0.9: 1255887.0000 - val_precision0.9: 0.7187 - val_recall0.9: 0.6029 - val_accuracy: 0.8118 - val_auc: 0.8811 - val_f1: 0.2772
    Epoch 2/20
    300/300 [==============================] - 109s 364ms/step - loss: 1.1355 - tp0.1: 9957479.0000 - fp0.1: 13357487.0000 - tn0.1: 53315960.0000 - fn0.1: 2012288.0000 - precision0.1: 0.4271 -
     recall0.1: 0.8319 - tp0.3: 8569227.0000 - fp0.3: 4466847.0000 - tn0.3: 62206584.0000 - fn0.3: 3400540.0000 - precision0.3: 0.6573 - recall0.3: 0.7159 - tp0.5: 7483729.0000 - fp0.5: 216675
    6.0000 - tn0.5: 64506688.0000 - fn0.5: 4486038.0000 - precision0.5: 0.7755 - recall0.5: 0.6252 - tp0.7: 6133208.0000 - fp0.7: 879463.0000 - tn0.7: 65793980.0000 - fn0.7: 5836559.0000 - pre
    cision0.7: 0.8746 - recall0.7: 0.5124 - tp0.9: 4221172.0000 - fp0.9: 179821.0000 - tn0.9: 66493600.0000 - fn0.9: 7748595.0000 - precision0.9: 0.9591 - recall0.9: 0.3527 - accuracy: 0.9154
    - auc: 0.8904 - f1: 0.2642 - val_loss: 1.1731 - val_tp0.1: 3054894.0000 - val_fp0.1: 9937096.0000 - val_tn0.1: 6560850.0000 - val_fn0.1: 107960.0000 - val_precision0.1: 0.2351 - val_recall
    0.1: 0.9659 - val_tp0.3: 2783981.0000 - val_fp0.3: 3337977.0000 - val_tn0.3: 13159969.0000 - val_fn0.3: 378873.0000 - val_precision0.3: 0.4548 - val_recall0.3: 0.8802 - val_tp0.5: 2388364.
    0000 - val_fp0.5: 1174330.0000 - val_tn0.5: 15323616.0000 - val_fn0.5: 774490.0000 - val_precision0.5: 0.6704 - val_recall0.5: 0.7551 - val_tp0.7: 1964516.0000 - val_fp0.7: 504808.0000 - v
    al_tn0.7: 15993138.0000 - val_fn0.7: 1198338.0000 - val_precision0.7: 0.7956 - val_recall0.7: 0.6211 - val_tp0.9: 1396619.0000 - val_fp0.9: 139367.0000 - val_tn0.9: 16358579.0000 - val_fn0
    .9: 1766235.0000 - val_precision0.9: 0.9093 - val_recall0.9: 0.4416 - val_accuracy: 0.9009 - val_auc: 0.9170 - val_f1: 0.2772
    Epoch 3/20
    300/300 [==============================] - 109s 365ms/step - loss: 1.1191 - tp0.1: 9992425.0000 - fp0.1: 11721660.0000 - tn0.1: 54951760.0000 - fn0.1: 1977342.0000 - precision0.1: 0.4602 -
     recall0.1: 0.8348 - tp0.3: 8827502.0000 - fp0.3: 4156590.0000 - tn0.3: 62516832.0000 - fn0.3: 3142265.0000 - precision0.3: 0.6799 - recall0.3: 0.7375 - tp0.5: 7896775.0000 - fp0.5: 216438
    4.0000 - tn0.5: 64509016.0000 - fn0.5: 4072992.0000 - precision0.5: 0.7849 - recall0.5: 0.6597 - tp0.7: 6592308.0000 - fp0.7: 940147.0000 - tn0.7: 65733296.0000 - fn0.7: 5377459.0000 - pre
    cision0.7: 0.8752 - recall0.7: 0.5507 - tp0.9: 4505055.0000 - fp0.9: 193708.0000 - tn0.9: 66479780.0000 - fn0.9: 7464712.0000 - precision0.9: 0.9588 - recall0.9: 0.3764 - accuracy: 0.9207
    - auc: 0.9016 - f1: 0.2642 - val_loss: 1.1653 - val_tp0.1: 2999133.0000 - val_fp0.1: 7651011.0000 - val_tn0.1: 8846935.0000 - val_fn0.1: 163721.0000 - val_precision0.1: 0.2816 - val_recall
    0.1: 0.9482 - val_tp0.3: 2753881.0000 - val_fp0.3: 2809322.0000 - val_tn0.3: 13688624.0000 - val_fn0.3: 408973.0000 - val_precision0.3: 0.4950 - val_recall0.3: 0.8707 - val_tp0.5: 2575607.
    0000 - val_fp0.5: 1604092.0000 - val_tn0.5: 14893854.0000 - val_fn0.5: 587247.0000 - val_precision0.5: 0.6162 - val_recall0.5: 0.8143 - val_tp0.7: 2325512.0000 - val_fp0.7: 840619.0000 - v
    al_tn0.7: 15657327.0000 - val_fn0.7: 837342.0000 - val_precision0.7: 0.7345 - val_recall0.7: 0.7353 - val_tp0.9: 1883413.0000 - val_fp0.9: 305362.0000 - val_tn0.9: 16192584.0000 - val_fn0.
    9: 1279441.0000 - val_precision0.9: 0.8605 - val_recall0.9: 0.5955 - val_accuracy: 0.8885 - val_auc: 0.9234 - val_f1: 0.2772
    Epoch 4/20
    300/300 [==============================] - 109s 364ms/step - loss: 0.7296 - tp0.1: 9976082.0000 - fp0.1: 10739149.0000 - tn0.1: 55934272.0000 - fn0.1: 1993685.0000 - precision0.1: 0.4816 -
     recall0.1: 0.8334 - tp0.3: 8926198.0000 - fp0.3: 4393527.0000 - tn0.3: 62279884.0000 - fn0.3: 3043569.0000 - precision0.3: 0.6701 - recall0.3: 0.7457 - tp0.5: 7905026.0000 - fp0.5: 217465
    0.0000 - tn0.5: 64498764.0000 - fn0.5: 4064741.0000 - precision0.5: 0.7843 - recall0.5: 0.6604 - tp0.7: 6608527.0000 - fp0.7: 943676.0000 - tn0.7: 65729772.0000 - fn0.7: 5361240.0000 - pre
    cision0.7: 0.8750 - recall0.7: 0.5521 - tp0.9: 4494816.0000 - fp0.9: 193397.0000 - tn0.9: 66480032.0000 - fn0.9: 7474951.0000 - precision0.9: 0.9587 - recall0.9: 0.3755 - accuracy: 0.9207
    - auc: 0.8902 - f1: 0.2642 - val_loss: 0.9724 - val_tp0.1: 2946822.0000 - val_fp0.1: 7011140.0000 - val_tn0.1: 9486806.0000 - val_fn0.1: 216032.0000 - val_precision0.1: 0.2959 - val_recall
    0.1: 0.9317 - val_tp0.3: 2828230.0000 - val_fp0.3: 4270134.0000 - val_tn0.3: 12227812.0000 - val_fn0.3: 334624.0000 - val_precision0.3: 0.3984 - val_recall0.3: 0.8942 - val_tp0.5: 2748329.
    0000 - val_fp0.5: 3138657.0000 - val_tn0.5: 13359289.0000 - val_fn0.5: 414525.0000 - val_precision0.5: 0.4668 - val_recall0.5: 0.8689 - val_tp0.7: 2589983.0000 - val_fp0.7: 1878374.0000 -
    val_tn0.7: 14619572.0000 - val_fn0.7: 572871.0000 - val_precision0.7: 0.5796 - val_recall0.7: 0.8189 - val_tp0.9: 2101517.0000 - val_fp0.9: 517482.0000 - val_tn0.9: 15980464.0000 - val_fn0
    .9: 1061337.0000 - val_precision0.9: 0.8024 - val_recall0.9: 0.6644 - val_accuracy: 0.8193 - val_auc: 0.9128 - val_f1: 0.2772
    Epoch 5/20
    300/300 [==============================] - 109s 363ms/step - loss: 0.6529 - tp0.1: 10122261.0000 - fp0.1: 9591155.0000 - tn0.1: 57082268.0000 - fn0.1: 1847506.0000 - precision0.1: 0.5135 -
     recall0.1: 0.8457 - tp0.3: 9144708.0000 - fp0.3: 3989412.0000 - tn0.3: 62684028.0000 - fn0.3: 2825059.0000 - precision0.3: 0.6963 - recall0.3: 0.7640 - tp0.5: 8259383.0000 - fp0.5: 210977
    9.0000 - tn0.5: 64563668.0000 - fn0.5: 3710384.0000 - precision0.5: 0.7965 - recall0.5: 0.6900 - tp0.7: 7094469.0000 - fp0.7: 988158.0000 - tn0.7: 65685280.0000 - fn0.7: 4875298.0000 - pre
    cision0.7: 0.8777 - recall0.7: 0.5927 - tp0.9: 4946279.0000 - fp0.9: 215942.0000 - tn0.9: 66457496.0000 - fn0.9: 7023488.0000 - precision0.9: 0.9582 - recall0.9: 0.4132 - accuracy: 0.9260
    - auc: 0.8999 - f1: 0.2642 - val_loss: 0.9432 - val_tp0.1: 3012100.0000 - val_fp0.1: 7946515.0000 - val_tn0.1: 8551431.0000 - val_fn0.1: 150754.0000 - val_precision0.1: 0.2749 - val_recall
    0.1: 0.9523 - val_tp0.3: 2894544.0000 - val_fp0.3: 4511247.0000 - val_tn0.3: 11986699.0000 - val_fn0.3: 268310.0000 - val_precision0.3: 0.3908 - val_recall0.3: 0.9152 - val_tp0.5: 2779750.
    0000 - val_fp0.5: 2798125.0000 - val_tn0.5: 13699821.0000 - val_fn0.5: 383104.0000 - val_precision0.5: 0.4984 - val_recall0.5: 0.8789 - val_tp0.7: 2629141.0000 - val_fp0.7: 1703471.0000 -
    val_tn0.7: 14794475.0000 - val_fn0.7: 533713.0000 - val_precision0.7: 0.6068 - val_recall0.7: 0.8313 - val_tp0.9: 2267860.0000 - val_fp0.9: 655578.0000 - val_tn0.9: 15842368.0000 - val_fn0
    .9: 894994.0000 - val_precision0.9: 0.7758 - val_recall0.9: 0.7170 - val_accuracy: 0.8382 - val_auc: 0.9243 - val_f1: 0.2772
    Epoch 6/20
    300/300 [==============================] - 109s 363ms/step - loss: 0.6248 - tp0.1: 10165542.0000 - fp0.1: 8993953.0000 - tn0.1: 57679492.0000 - fn0.1: 1804225.0000 - precision0.1: 0.5306 -
     recall0.1: 0.8493 - tp0.3: 9220264.0000 - fp0.3: 3729094.0000 - tn0.3: 62944360.0000 - fn0.3: 2749503.0000 - precision0.3: 0.7120 - recall0.3: 0.7703 - tp0.5: 8427138.0000 - fp0.5: 202054
    3.0000 - tn0.5: 64652888.0000 - fn0.5: 3542629.0000 - precision0.5: 0.8066 - recall0.5: 0.7040 - tp0.7: 7328827.0000 - fp0.7: 939211.0000 - tn0.7: 65734200.0000 - fn0.7: 4640940.0000 - pre
    cision0.7: 0.8864 - recall0.7: 0.6123 - tp0.9: 5167941.0000 - fp0.9: 189519.0000 - tn0.9: 66483928.0000 - fn0.9: 6801826.0000 - precision0.9: 0.9646 - recall0.9: 0.4317 - accuracy: 0.9293
    - auc: 0.9044 - f1: 0.2642 - val_loss: 0.5511 - val_tp0.1: 2844793.0000 - val_fp0.1: 2570394.0000 - val_tn0.1: 13927552.0000 - val_fn0.1: 318061.0000 - val_precision0.1: 0.5253 - val_recal
    l0.1: 0.8994 - val_tp0.3: 2684206.0000 - val_fp0.3: 1367923.0000 - val_tn0.3: 15130023.0000 - val_fn0.3: 478648.0000 - val_precision0.3: 0.6624 - val_recall0.3: 0.8487 - val_tp0.5: 2475186
    .0000 - val_fp0.5: 730745.0000 - val_tn0.5: 15767201.0000 - val_fn0.5: 687668.0000 - val_precision0.5: 0.7721 - val_recall0.5: 0.7826 - val_tp0.7: 2218398.0000 - val_fp0.7: 388178.0000 - v
    al_tn0.7: 16109768.0000 - val_fn0.7: 944456.0000 - val_precision0.7: 0.8511 - val_recall0.7: 0.7014 - val_tp0.9: 1713349.0000 - val_fp0.9: 114270.0000 - val_tn0.9: 16383676.0000 - val_fn0.
    9: 1449505.0000 - val_precision0.9: 0.9375 - val_recall0.9: 0.5417 - val_accuracy: 0.9279 - val_auc: 0.9285 - val_f1: 0.2772
    Epoch 7/20
    300/300 [==============================] - 109s 364ms/step - loss: 0.6064 - tp0.1: 10229897.0000 - fp0.1: 8897750.0000 - tn0.1: 57775700.0000 - fn0.1: 1739870.0000 - precision0.1: 0.5348 -
     recall0.1: 0.8546 - tp0.3: 9283055.0000 - fp0.3: 3514175.0000 - tn0.3: 63159248.0000 - fn0.3: 2686712.0000 - precision0.3: 0.7254 - recall0.3: 0.7755 - tp0.5: 8533348.0000 - fp0.5: 192231
    9.0000 - tn0.5: 64751124.0000 - fn0.5: 3436419.0000 - precision0.5: 0.8161 - recall0.5: 0.7129 - tp0.7: 7491425.0000 - fp0.7: 917374.0000 - tn0.7: 65756056.0000 - fn0.7: 4478342.0000 - pre
    cision0.7: 0.8909 - recall0.7: 0.6259 - tp0.9: 5439611.0000 - fp0.9: 210848.0000 - tn0.9: 66462588.0000 - fn0.9: 6530156.0000 - precision0.9: 0.9627 - recall0.9: 0.4544 - accuracy: 0.9319
    - auc: 0.9082 - f1: 0.2642 - val_loss: 0.6010 - val_tp0.1: 2934541.0000 - val_fp0.1: 3995941.0000 - val_tn0.1: 12502005.0000 - val_fn0.1: 228313.0000 - val_precision0.1: 0.4234 - val_recal
    l0.1: 0.9278 - val_tp0.3: 2764037.0000 - val_fp0.3: 1908942.0000 - val_tn0.3: 14589004.0000 - val_fn0.3: 398817.0000 - val_precision0.3: 0.5915 - val_recall0.3: 0.8739 - val_tp0.5: 2611207
    .0000 - val_fp0.5: 1127029.0000 - val_tn0.5: 15370917.0000 - val_fn0.5: 551647.0000 - val_precision0.5: 0.6985 - val_recall0.5: 0.8256 - val_tp0.7: 2388708.0000 - val_fp0.7: 602484.0000 -
    val_tn0.7: 15895462.0000 - val_fn0.7: 774146.0000 - val_precision0.7: 0.7986 - val_recall0.7: 0.7552 - val_tp0.9: 1910700.0000 - val_fp0.9: 183899.0000 - val_tn0.9: 16314047.0000 - val_fn0
    .9: 1252154.0000 - val_precision0.9: 0.9122 - val_recall0.9: 0.6041 - val_accuracy: 0.9146 - val_auc: 0.9353 - val_f1: 0.2772
    Epoch 8/20
    300/300 [==============================] - 109s 364ms/step - loss: 0.5966 - tp0.1: 10258117.0000 - fp0.1: 8820090.0000 - tn0.1: 57853336.0000 - fn0.1: 1711650.0000 - precision0.1: 0.5377 -
     recall0.1: 0.8570 - tp0.3: 9332270.0000 - fp0.3: 3494563.0000 - tn0.3: 63178884.0000 - fn0.3: 2637497.0000 - precision0.3: 0.7276 - recall0.3: 0.7797 - tp0.5: 8607177.0000 - fp0.5: 193241
    9.0000 - tn0.5: 64741044.0000 - fn0.5: 3362590.0000 - precision0.5: 0.8167 - recall0.5: 0.7191 - tp0.7: 7560035.0000 - fp0.7: 907227.0000 - tn0.7: 65766188.0000 - fn0.7: 4409732.0000 - pre
    cision0.7: 0.8929 - recall0.7: 0.6316 - tp0.9: 5489005.0000 - fp0.9: 193323.0000 - tn0.9: 66480128.0000 - fn0.9: 6480762.0000 - precision0.9: 0.9660 - recall0.9: 0.4586 - accuracy: 0.9327
    - auc: 0.9098 - f1: 0.2642 - val_loss: 0.5360 - val_tp0.1: 2840705.0000 - val_fp0.1: 2432393.0000 - val_tn0.1: 14065553.0000 - val_fn0.1: 322149.0000 - val_precision0.1: 0.5387 - val_recal
    l0.1: 0.8981 - val_tp0.3: 2541562.0000 - val_fp0.3: 814687.0000 - val_tn0.3: 15683259.0000 - val_fn0.3: 621292.0000 - val_precision0.3: 0.7573 - val_recall0.3: 0.8036 - val_tp0.5: 2332487.
    0000 - val_fp0.5: 450050.0000 - val_tn0.5: 16047896.0000 - val_fn0.5: 830367.0000 - val_precision0.5: 0.8383 - val_recall0.5: 0.7375 - val_tp0.7: 2027801.0000 - val_fp0.7: 205102.0000 - va
    l_tn0.7: 16292844.0000 - val_fn0.7: 1135053.0000 - val_precision0.7: 0.9081 - val_recall0.7: 0.6411 - val_tp0.9: 1478341.0000 - val_fp0.9: 39136.0000 - val_tn0.9: 16458810.0000 - val_fn0.9
    : 1684513.0000 - val_precision0.9: 0.9742 - val_recall0.9: 0.4674 - val_accuracy: 0.9349 - val_auc: 0.9309 - val_f1: 0.2772
    Epoch 9/20
    300/300 [==============================] - 109s 363ms/step - loss: 0.5815 - tp0.1: 10317877.0000 - fp0.1: 8787663.0000 - tn0.1: 57885784.0000 - fn0.1: 1651890.0000 - precision0.1: 0.5400 -
     recall0.1: 0.8620 - tp0.3: 9391866.0000 - fp0.3: 3385982.0000 - tn0.3: 63287432.0000 - fn0.3: 2577901.0000 - precision0.3: 0.7350 - recall0.3: 0.7846 - tp0.5: 8683886.0000 - fp0.5: 185537
    9.0000 - tn0.5: 64818064.0000 - fn0.5: 3285881.0000 - precision0.5: 0.8240 - recall0.5: 0.7255 - tp0.7: 7664640.0000 - fp0.7: 872040.0000 - tn0.7: 65801404.0000 - fn0.7: 4305127.0000 - pre
    cision0.7: 0.8978 - recall0.7: 0.6403 - tp0.9: 5645583.0000 - fp0.9: 192263.0000 - tn0.9: 66481188.0000 - fn0.9: 6324184.0000 - precision0.9: 0.9671 - recall0.9: 0.4717 - accuracy: 0.9346
    - auc: 0.9130 - f1: 0.2642 - val_loss: 0.5508 - val_tp0.1: 2890310.0000 - val_fp0.1: 3133203.0000 - val_tn0.1: 13364743.0000 - val_fn0.1: 272544.0000 - val_precision0.1: 0.4798 - val_recal
    l0.1: 0.9138 - val_tp0.3: 2582821.0000 - val_fp0.3: 941605.0000 - val_tn0.3: 15556341.0000 - val_fn0.3: 580033.0000 - val_precision0.3: 0.7328 - val_recall0.3: 0.8166 - val_tp0.5: 2342586.
    0000 - val_fp0.5: 468856.0000 - val_tn0.5: 16029090.0000 - val_fn0.5: 820268.0000 - val_precision0.5: 0.8332 - val_recall0.5: 0.7407 - val_tp0.7: 2000631.0000 - val_fp0.7: 196312.0000 - va
    l_tn0.7: 16301634.0000 - val_fn0.7: 1162223.0000 - val_precision0.7: 0.9106 - val_recall0.7: 0.6325 - val_tp0.9: 1393863.0000 - val_fp0.9: 35445.0000 - val_tn0.9: 16462501.0000 - val_fn0.9
    : 1768991.0000 - val_precision0.9: 0.9752 - val_recall0.9: 0.4407 - val_accuracy: 0.9344 - val_auc: 0.9339 - val_f1: 0.2772
    Epoch 10/20
    300/300 [==============================] - 109s 363ms/step - loss: 0.5742 - tp0.1: 10342521.0000 - fp0.1: 8755470.0000 - tn0.1: 57917940.0000 - fn0.1: 1627246.0000 - precision0.1: 0.5416 -
     recall0.1: 0.8641 - tp0.3: 9412910.0000 - fp0.3: 3294713.0000 - tn0.3: 63378728.0000 - fn0.3: 2556857.0000 - precision0.3: 0.7407 - recall0.3: 0.7864 - tp0.5: 8709465.0000 - fp0.5: 179218
    3.0000 - tn0.5: 64881220.0000 - fn0.5: 3260302.0000 - precision0.5: 0.8293 - recall0.5: 0.7276 - tp0.7: 7716001.0000 - fp0.7: 846147.0000 - tn0.7: 65827296.0000 - fn0.7: 4253766.0000 - pre
    cision0.7: 0.9012 - recall0.7: 0.6446 - tp0.9: 5786134.0000 - fp0.9: 193748.0000 - tn0.9: 66479688.0000 - fn0.9: 6183633.0000 - precision0.9: 0.9676 - recall0.9: 0.4834 - accuracy: 0.9358
    - auc: 0.9142 - f1: 0.2642 - val_loss: 0.5714 - val_tp0.1: 2695380.0000 - val_fp0.1: 1362499.0000 - val_tn0.1: 15135447.0000 - val_fn0.1: 467474.0000 - val_precision0.1: 0.6642 - val_recal
    l0.1: 0.8522 - val_tp0.3: 2331121.0000 - val_fp0.3: 439292.0000 - val_tn0.3: 16058654.0000 - val_fn0.3: 831733.0000 - val_precision0.3: 0.8414 - val_recall0.3: 0.7370 - val_tp0.5: 2008752.
    0000 - val_fp0.5: 188756.0000 - val_tn0.5: 16309190.0000 - val_fn0.5: 1154102.0000 - val_precision0.5: 0.9141 - val_recall0.5: 0.6351 - val_tp0.7: 1582146.0000 - val_fp0.7: 59586.0000 - va
    l_tn0.7: 16438360.0000 - val_fn0.7: 1580708.0000 - val_precision0.7: 0.9637 - val_recall0.7: 0.5002 - val_tp0.9: 896176.0000 - val_fp0.9: 4415.0000 - val_tn0.9: 16493531.0000 - val_fn0.9:
    2266678.0000 - val_precision0.9: 0.9951 - val_recall0.9: 0.2833 - val_accuracy: 0.9317 - val_auc: 0.9152 - val_f1: 0.2772
    Epoch 11/20
    300/300 [==============================] - 109s 364ms/step - loss: 0.5695 - tp0.1: 10353763.0000 - fp0.1: 8713907.0000 - tn0.1: 57959528.0000 - fn0.1: 1616004.0000 - precision0.1: 0.5430 -
     recall0.1: 0.8650 - tp0.3: 9444896.0000 - fp0.3: 3321969.0000 - tn0.3: 63351480.0000 - fn0.3: 2524871.0000 - precision0.3: 0.7398 - recall0.3: 0.7891 - tp0.5: 8747813.0000 - fp0.5: 180398
    4.0000 - tn0.5: 64869452.0000 - fn0.5: 3221954.0000 - precision0.5: 0.8290 - recall0.5: 0.7308 - tp0.7: 7782574.0000 - fp0.7: 853915.0000 - tn0.7: 65819536.0000 - fn0.7: 4187193.0000 - pre
    cision0.7: 0.9011 - recall0.7: 0.6502 - tp0.9: 5870371.0000 - fp0.9: 194465.0000 - tn0.9: 66479000.0000 - fn0.9: 6099396.0000 - precision0.9: 0.9679 - recall0.9: 0.4904 - accuracy: 0.9361
    - auc: 0.9151 - f1: 0.2642 - val_loss: 0.5577 - val_tp0.1: 2892745.0000 - val_fp0.1: 3143327.0000 - val_tn0.1: 13354619.0000 - val_fn0.1: 270109.0000 - val_precision0.1: 0.4792 - val_recal
    l0.1: 0.9146 - val_tp0.3: 2706166.0000 - val_fp0.3: 1385967.0000 - val_tn0.3: 15111979.0000 - val_fn0.3: 456688.0000 - val_precision0.3: 0.6613 - val_recall0.3: 0.8556 - val_tp0.5: 2567444
    .0000 - val_fp0.5: 857066.0000 - val_tn0.5: 15640880.0000 - val_fn0.5: 595410.0000 - val_precision0.5: 0.7497 - val_recall0.5: 0.8117 - val_tp0.7: 2361482.0000 - val_fp0.7: 478495.0000 - v
    al_tn0.7: 16019451.0000 - val_fn0.7: 801372.0000 - val_precision0.7: 0.8315 - val_recall0.7: 0.7466 - val_tp0.9: 1980797.0000 - val_fp0.9: 176419.0000 - val_tn0.9: 16321527.0000 - val_fn0.
    9: 1182057.0000 - val_precision0.9: 0.9182 - val_recall0.9: 0.6263 - val_accuracy: 0.9261 - val_auc: 0.9342 - val_f1: 0.2772
    Epoch 12/20
    300/300 [==============================] - 109s 364ms/step - loss: 0.5631 - tp0.1: 10359789.0000 - fp0.1: 8488876.0000 - tn0.1: 58184560.0000 - fn0.1: 1609978.0000 - precision0.1: 0.5496 -
     recall0.1: 0.8655 - tp0.3: 9469292.0000 - fp0.3: 3249710.0000 - tn0.3: 63423736.0000 - fn0.3: 2500475.0000 - precision0.3: 0.7445 - recall0.3: 0.7911 - tp0.5: 8773908.0000 - fp0.5: 173654
    4.0000 - tn0.5: 64936880.0000 - fn0.5: 3195859.0000 - precision0.5: 0.8348 - recall0.5: 0.7330 - tp0.7: 7799121.0000 - fp0.7: 807785.0000 - tn0.7: 65865644.0000 - fn0.7: 4170646.0000 - pre
    cision0.7: 0.9061 - recall0.7: 0.6516 - tp0.9: 5889535.0000 - fp0.9: 186306.0000 - tn0.9: 66487124.0000 - fn0.9: 6080232.0000 - precision0.9: 0.9693 - recall0.9: 0.4920 - accuracy: 0.9373
    - auc: 0.9159 - f1: 0.2642 - val_loss: 0.5163 - val_tp0.1: 2823752.0000 - val_fp0.1: 2094434.0000 - val_tn0.1: 14403512.0000 - val_fn0.1: 339102.0000 - val_precision0.1: 0.5741 - val_recal
    l0.1: 0.8928 - val_tp0.3: 2583025.0000 - val_fp0.3: 825425.0000 - val_tn0.3: 15672521.0000 - val_fn0.3: 579829.0000 - val_precision0.3: 0.7578 - val_recall0.3: 0.8167 - val_tp0.5: 2379325.
    0000 - val_fp0.5: 447908.0000 - val_tn0.5: 16050038.0000 - val_fn0.5: 783529.0000 - val_precision0.5: 0.8416 - val_recall0.5: 0.7523 - val_tp0.7: 2087728.0000 - val_fp0.7: 207419.0000 - va
    l_tn0.7: 16290527.0000 - val_fn0.7: 1075126.0000 - val_precision0.7: 0.9096 - val_recall0.7: 0.6601 - val_tp0.9: 1527766.0000 - val_fp0.9: 40879.0000 - val_tn0.9: 16457067.0000 - val_fn0.9
    : 1635088.0000 - val_precision0.9: 0.9739 - val_recall0.9: 0.4830 - val_accuracy: 0.9374 - val_auc: 0.9303 - val_f1: 0.2772
    Epoch 13/20
    300/300 [==============================] - 109s 364ms/step - loss: 0.5583 - tp0.1: 10369243.0000 - fp0.1: 8417163.0000 - tn0.1: 58256252.0000 - fn0.1: 1600524.0000 - precision0.1: 0.5520 -
     recall0.1: 0.8663 - tp0.3: 9495979.0000 - fp0.3: 3288261.0000 - tn0.3: 63385168.0000 - fn0.3: 2473788.0000 - precision0.3: 0.7428 - recall0.3: 0.7933 - tp0.5: 8802292.0000 - fp0.5: 176251
    9.0000 - tn0.5: 64910900.0000 - fn0.5: 3167475.0000 - precision0.5: 0.8332 - recall0.5: 0.7354 - tp0.7: 7853085.0000 - fp0.7: 831023.0000 - tn0.7: 65842424.0000 - fn0.7: 4116682.0000 - pre
    cision0.7: 0.9043 - recall0.7: 0.6561 - tp0.9: 5967523.0000 - fp0.9: 191510.0000 - tn0.9: 66481912.0000 - fn0.9: 6002244.0000 - precision0.9: 0.9689 - recall0.9: 0.4985 - accuracy: 0.9373
    - auc: 0.9166 - f1: 0.2642 - val_loss: 0.5155 - val_tp0.1: 2810372.0000 - val_fp0.1: 1944105.0000 - val_tn0.1: 14553841.0000 - val_fn0.1: 352482.0000 - val_precision0.1: 0.5911 - val_recal
    l0.1: 0.8886 - val_tp0.3: 2550568.0000 - val_fp0.3: 737116.0000 - val_tn0.3: 15760830.0000 - val_fn0.3: 612286.0000 - val_precision0.3: 0.7758 - val_recall0.3: 0.8064 - val_tp0.5: 2338331.
    0000 - val_fp0.5: 398618.0000 - val_tn0.5: 16099328.0000 - val_fn0.5: 824523.0000 - val_precision0.5: 0.8544 - val_recall0.5: 0.7393 - val_tp0.7: 2056023.0000 - val_fp0.7: 191281.0000 - va
    l_tn0.7: 16306665.0000 - val_fn0.7: 1106831.0000 - val_precision0.7: 0.9149 - val_recall0.7: 0.6501 - val_tp0.9: 1536052.0000 - val_fp0.9: 42617.0000 - val_tn0.9: 16455329.0000 - val_fn0.9
    : 1626802.0000 - val_precision0.9: 0.9730 - val_recall0.9: 0.4857 - val_accuracy: 0.9378 - val_auc: 0.9292 - val_f1: 0.2772
    Epoch 14/20
    300/300 [==============================] - 109s 364ms/step - loss: 0.5545 - tp0.1: 10404673.0000 - fp0.1: 8594836.0000 - tn0.1: 58078576.0000 - fn0.1: 1565094.0000 - precision0.1: 0.5476 -
     recall0.1: 0.8692 - tp0.3: 9502698.0000 - fp0.3: 3265228.0000 - tn0.3: 63408216.0000 - fn0.3: 2467069.0000 - precision0.3: 0.7443 - recall0.3: 0.7939 - tp0.5: 8806002.0000 - fp0.5: 173452
    7.0000 - tn0.5: 64938912.0000 - fn0.5: 3163765.0000 - precision0.5: 0.8354 - recall0.5: 0.7357 - tp0.7: 7871330.0000 - fp0.7: 811976.0000 - tn0.7: 65861468.0000 - fn0.7: 4098437.0000 - pre
    cision0.7: 0.9065 - recall0.7: 0.6576 - tp0.9: 5995175.0000 - fp0.9: 182094.0000 - tn0.9: 66491348.0000 - fn0.9: 5974592.0000 - precision0.9: 0.9705 - recall0.9: 0.5009 - accuracy: 0.9377
    - auc: 0.9180 - f1: 0.2642 - val_loss: 0.5147 - val_tp0.1: 2829492.0000 - val_fp0.1: 2130176.0000 - val_tn0.1: 14367770.0000 - val_fn0.1: 333362.0000 - val_precision0.1: 0.5705 - val_recal
    l0.1: 0.8946 - val_tp0.3: 2583127.0000 - val_fp0.3: 813224.0000 - val_tn0.3: 15684722.0000 - val_fn0.3: 579727.0000 - val_precision0.3: 0.7606 - val_recall0.3: 0.8167 - val_tp0.5: 2383747.
    0000 - val_fp0.5: 446660.0000 - val_tn0.5: 16051286.0000 - val_fn0.5: 779107.0000 - val_precision0.5: 0.8422 - val_recall0.5: 0.7537 - val_tp0.7: 2116906.0000 - val_fp0.7: 219992.0000 - va
    l_tn0.7: 16277954.0000 - val_fn0.7: 1045948.0000 - val_precision0.7: 0.9059 - val_recall0.7: 0.6693 - val_tp0.9: 1607900.0000 - val_fp0.9: 51317.0000 - val_tn0.9: 16446629.0000 - val_fn0.9
    : 1554954.0000 - val_precision0.9: 0.9691 - val_recall0.9: 0.5084 - val_accuracy: 0.9377 - val_auc: 0.9312 - val_f1: 0.2772
    Epoch 15/20
    300/300 [==============================] - 109s 364ms/step - loss: 0.5536 - tp0.1: 10410305.0000 - fp0.1: 8632096.0000 - tn0.1: 58041320.0000 - fn0.1: 1559462.0000 - precision0.1: 0.5467 -
     recall0.1: 0.8697 - tp0.3: 9510984.0000 - fp0.3: 3242903.0000 - tn0.3: 63430520.0000 - fn0.3: 2458783.0000 - precision0.3: 0.7457 - recall0.3: 0.7946 - tp0.5: 8812511.0000 - fp0.5: 170138
    1.0000 - tn0.5: 64972056.0000 - fn0.5: 3157256.0000 - precision0.5: 0.8382 - recall0.5: 0.7362 - tp0.7: 7877931.0000 - fp0.7: 793039.0000 - tn0.7: 65880400.0000 - fn0.7: 4091836.0000 - pre
    cision0.7: 0.9085 - recall0.7: 0.6582 - tp0.9: 5978990.0000 - fp0.9: 174678.0000 - tn0.9: 66498776.0000 - fn0.9: 5990777.0000 - precision0.9: 0.9716 - recall0.9: 0.4995 - accuracy: 0.9382
    - auc: 0.9183 - f1: 0.2642 - val_loss: 0.5193 - val_tp0.1: 2813989.0000 - val_fp0.1: 2032257.0000 - val_tn0.1: 14465689.0000 - val_fn0.1: 348865.0000 - val_precision0.1: 0.5807 - val_recal
    l0.1: 0.8897 - val_tp0.3: 2577154.0000 - val_fp0.3: 819794.0000 - val_tn0.3: 15678152.0000 - val_fn0.3: 585700.0000 - val_precision0.3: 0.7587 - val_recall0.3: 0.8148 - val_tp0.5: 2385705.
    0000 - val_fp0.5: 459176.0000 - val_tn0.5: 16038770.0000 - val_fn0.5: 777149.0000 - val_precision0.5: 0.8386 - val_recall0.5: 0.7543 - val_tp0.7: 2130654.0000 - val_fp0.7: 230334.0000 - va
    l_tn0.7: 16267612.0000 - val_fn0.7: 1032200.0000 - val_precision0.7: 0.9024 - val_recall0.7: 0.6736 - val_tp0.9: 1640662.0000 - val_fp0.9: 55692.0000 - val_tn0.9: 16442254.0000 - val_fn0.9
    : 1522192.0000 - val_precision0.9: 0.9672 - val_recall0.9: 0.5187 - val_accuracy: 0.9371 - val_auc: 0.9291 - val_f1: 0.2772
    Epoch 16/20
    300/300 [==============================] - 109s 364ms/step - loss: 0.5574 - tp0.1: 10366848.0000 - fp0.1: 8351245.0000 - tn0.1: 58322184.0000 - fn0.1: 1602919.0000 - precision0.1: 0.5538 -
     recall0.1: 0.8661 - tp0.3: 9482187.0000 - fp0.3: 3195743.0000 - tn0.3: 63477668.0000 - fn0.3: 2487580.0000 - precision0.3: 0.7479 - recall0.3: 0.7922 - tp0.5: 8790251.0000 - fp0.5: 169760
    0.0000 - tn0.5: 64975828.0000 - fn0.5: 3179516.0000 - precision0.5: 0.8381 - recall0.5: 0.7344 - tp0.7: 7865829.0000 - fp0.7: 803359.0000 - tn0.7: 65870080.0000 - fn0.7: 4103938.0000 - pre
    cision0.7: 0.9073 - recall0.7: 0.6571 - tp0.9: 6012200.0000 - fp0.9: 181018.0000 - tn0.9: 66492408.0000 - fn0.9: 5957567.0000 - precision0.9: 0.9708 - recall0.9: 0.5023 - accuracy: 0.9380
    - auc: 0.9168 - f1: 0.2642 - val_loss: 0.5190 - val_tp0.1: 2815670.0000 - val_fp0.1: 2047710.0000 - val_tn0.1: 14450236.0000 - val_fn0.1: 347184.0000 - val_precision0.1: 0.5790 - val_recal
    l0.1: 0.8902 - val_tp0.3: 2576965.0000 - val_fp0.3: 816090.0000 - val_tn0.3: 15681856.0000 - val_fn0.3: 585889.0000 - val_precision0.3: 0.7595 - val_recall0.3: 0.8148 - val_tp0.5: 2382191.
    0000 - val_fp0.5: 452866.0000 - val_tn0.5: 16045080.0000 - val_fn0.5: 780663.0000 - val_precision0.5: 0.8403 - val_recall0.5: 0.7532 - val_tp0.7: 2122711.0000 - val_fp0.7: 224345.0000 - va
    l_tn0.7: 16273601.0000 - val_fn0.7: 1040143.0000 - val_precision0.7: 0.9044 - val_recall0.7: 0.6711 - val_tp0.9: 1623419.0000 - val_fp0.9: 52719.0000 - val_tn0.9: 16445227.0000 - val_fn0.9
    : 1539435.0000 - val_precision0.9: 0.9685 - val_recall0.9: 0.5133 - val_accuracy: 0.9373 - val_auc: 0.9293 - val_f1: 0.2772
    Epoch 17/20
    300/300 [==============================] - 109s 364ms/step - loss: 0.5518 - tp0.1: 10385606.0000 - fp0.1: 8322065.0000 - tn0.1: 58351360.0000 - fn0.1: 1584161.0000 - precision0.1: 0.5552 -
     recall0.1: 0.8677 - tp0.3: 9501942.0000 - fp0.3: 3183126.0000 - tn0.3: 63490304.0000 - fn0.3: 2467825.0000 - precision0.3: 0.7491 - recall0.3: 0.7938 - tp0.5: 8803116.0000 - fp0.5: 167708
    6.0000 - tn0.5: 64996332.0000 - fn0.5: 3166651.0000 - precision0.5: 0.8400 - recall0.5: 0.7354 - tp0.7: 7870926.0000 - fp0.7: 783130.0000 - tn0.7: 65890296.0000 - fn0.7: 4098841.0000 - pre
    cision0.7: 0.9095 - recall0.7: 0.6576 - tp0.9: 5981676.0000 - fp0.9: 175979.0000 - tn0.9: 66497456.0000 - fn0.9: 5988091.0000 - precision0.9: 0.9714 - recall0.9: 0.4997 - accuracy: 0.9384
    - auc: 0.9178 - f1: 0.2642 - val_loss: 0.5176 - val_tp0.1: 2814391.0000 - val_fp0.1: 2019842.0000 - val_tn0.1: 14478104.0000 - val_fn0.1: 348463.0000 - val_precision0.1: 0.5822 - val_recal
    l0.1: 0.8898 - val_tp0.3: 2575426.0000 - val_fp0.3: 804457.0000 - val_tn0.3: 15693489.0000 - val_fn0.3: 587428.0000 - val_precision0.3: 0.7620 - val_recall0.3: 0.8143 - val_tp0.5: 2377184.
    0000 - val_fp0.5: 443147.0000 - val_tn0.5: 16054799.0000 - val_fn0.5: 785670.0000 - val_precision0.5: 0.8429 - val_recall0.5: 0.7516 - val_tp0.7: 2111101.0000 - val_fp0.7: 215747.0000 - va
    l_tn0.7: 16282199.0000 - val_fn0.7: 1051753.0000 - val_precision0.7: 0.9073 - val_recall0.7: 0.6675 - val_tp0.9: 1598659.0000 - val_fp0.9: 48117.0000 - val_tn0.9: 16449829.0000 - val_fn0.9
    : 1564195.0000 - val_precision0.9: 0.9708 - val_recall0.9: 0.5054 - val_accuracy: 0.9375 - val_auc: 0.9293 - val_f1: 0.2772
    Epoch 18/20
    300/300 [==============================] - 109s 365ms/step - loss: 0.5527 - tp0.1: 10394599.0000 - fp0.1: 8422625.0000 - tn0.1: 58250828.0000 - fn0.1: 1575168.0000 - precision0.1: 0.5524 -
     recall0.1: 0.8684 - tp0.3: 9515719.0000 - fp0.3: 3245634.0000 - tn0.3: 63427764.0000 - fn0.3: 2454048.0000 - precision0.3: 0.7457 - recall0.3: 0.7950 - tp0.5: 8829115.0000 - fp0.5: 172580
    7.0000 - tn0.5: 64947624.0000 - fn0.5: 3140652.0000 - precision0.5: 0.8365 - recall0.5: 0.7376 - tp0.7: 7902699.0000 - fp0.7: 815002.0000 - tn0.7: 65858428.0000 - fn0.7: 4067068.0000 - pre
    cision0.7: 0.9065 - recall0.7: 0.6602 - tp0.9: 5992723.0000 - fp0.9: 181617.0000 - tn0.9: 66491804.0000 - fn0.9: 5977044.0000 - precision0.9: 0.9706 - recall0.9: 0.5007 - accuracy: 0.9381
    - auc: 0.9180 - f1: 0.2642 - val_loss: 0.5170 - val_tp0.1: 2818280.0000 - val_fp0.1: 2051127.0000 - val_tn0.1: 14446819.0000 - val_fn0.1: 344574.0000 - val_precision0.1: 0.5788 - val_recal
    l0.1: 0.8911 - val_tp0.3: 2582361.0000 - val_fp0.3: 822058.0000 - val_tn0.3: 15675888.0000 - val_fn0.3: 580493.0000 - val_precision0.3: 0.7585 - val_recall0.3: 0.8165 - val_tp0.5: 2386572.
    0000 - val_fp0.5: 453718.0000 - val_tn0.5: 16044228.0000 - val_fn0.5: 776282.0000 - val_precision0.5: 0.8403 - val_recall0.5: 0.7546 - val_tp0.7: 2123293.0000 - val_fp0.7: 221964.0000 - va
    l_tn0.7: 16275982.0000 - val_fn0.7: 1039561.0000 - val_precision0.7: 0.9054 - val_recall0.7: 0.6713 - val_tp0.9: 1613816.0000 - val_fp0.9: 50399.0000 - val_tn0.9: 16447547.0000 - val_fn0.9
    : 1549038.0000 - val_precision0.9: 0.9697 - val_recall0.9: 0.5102 - val_accuracy: 0.9374 - val_auc: 0.9297 - val_f1: 0.2772
    Epoch 19/20
    300/300 [==============================] - 109s 364ms/step - loss: 0.5502 - tp0.1: 10400846.0000 - fp0.1: 8405890.0000 - tn0.1: 58267520.0000 - fn0.1: 1568921.0000 - precision0.1: 0.5530 -
     recall0.1: 0.8689 - tp0.3: 9521394.0000 - fp0.3: 3231081.0000 - tn0.3: 63442368.0000 - fn0.3: 2448373.0000 - precision0.3: 0.7466 - recall0.3: 0.7955 - tp0.5: 8830551.0000 - fp0.5: 170937
    1.0000 - tn0.5: 64964080.0000 - fn0.5: 3139216.0000 - precision0.5: 0.8378 - recall0.5: 0.7377 - tp0.7: 7900201.0000 - fp0.7: 803691.0000 - tn0.7: 65869728.0000 - fn0.7: 4069566.0000 - pre
    cision0.7: 0.9077 - recall0.7: 0.6600 - tp0.9: 5995260.0000 - fp0.9: 181599.0000 - tn0.9: 66491848.0000 - fn0.9: 5974507.0000 - precision0.9: 0.9706 - recall0.9: 0.5009 - accuracy: 0.9383
    - auc: 0.9184 - f1: 0.2642 - val_loss: 0.5175 - val_tp0.1: 2819063.0000 - val_fp0.1: 2064294.0000 - val_tn0.1: 14433652.0000 - val_fn0.1: 343791.0000 - val_precision0.1: 0.5773 - val_recal
    l0.1: 0.8913 - val_tp0.3: 2584585.0000 - val_fp0.3: 828069.0000 - val_tn0.3: 15669877.0000 - val_fn0.3: 578269.0000 - val_precision0.3: 0.7574 - val_recall0.3: 0.8172 - val_tp0.5: 2390134.
    0000 - val_fp0.5: 458519.0000 - val_tn0.5: 16039427.0000 - val_fn0.5: 772720.0000 - val_precision0.5: 0.8390 - val_recall0.5: 0.7557 - val_tp0.7: 2128769.0000 - val_fp0.7: 225470.0000 - va
    l_tn0.7: 16272476.0000 - val_fn0.7: 1034085.0000 - val_precision0.7: 0.9042 - val_recall0.7: 0.6731 - val_tp0.9: 1617794.0000 - val_fp0.9: 50968.0000 - val_tn0.9: 16446978.0000 - val_fn0.9
    : 1545060.0000 - val_precision0.9: 0.9695 - val_recall0.9: 0.5115 - val_accuracy: 0.9374 - val_auc: 0.9298 - val_f1: 0.2772
    Epoch 20/20
    300/300 [==============================] - 109s 364ms/step - loss: 0.5540 - tp0.1: 10388304.0000 - fp0.1: 8427637.0000 - tn0.1: 58245784.0000 - fn0.1: 1581463.0000 - precision0.1: 0.5521 -
     recall0.1: 0.8679 - tp0.3: 9510729.0000 - fp0.3: 3244455.0000 - tn0.3: 63429016.0000 - fn0.3: 2459038.0000 - precision0.3: 0.7456 - recall0.3: 0.7946 - tp0.5: 8826300.0000 - fp0.5: 173006
    6.0000 - tn0.5: 64943372.0000 - fn0.5: 3143467.0000 - precision0.5: 0.8361 - recall0.5: 0.7374 - tp0.7: 7907765.0000 - fp0.7: 815687.0000 - tn0.7: 65857736.0000 - fn0.7: 4062002.0000 - pre
    cision0.7: 0.9065 - recall0.7: 0.6606 - tp0.9: 6007828.0000 - fp0.9: 181663.0000 - tn0.9: 66491748.0000 - fn0.9: 5961939.0000 - precision0.9: 0.9706 - recall0.9: 0.5019 - accuracy: 0.9380
    - auc: 0.9177 - f1: 0.2642 - val_loss: 0.5180 - val_tp0.1: 2817068.0000 - val_fp0.1: 2045853.0000 - val_tn0.1: 14452093.0000 - val_fn0.1: 345786.0000 - val_precision0.1: 0.5793 - val_recal
    l0.1: 0.8907 - val_tp0.3: 2578834.0000 - val_fp0.3: 815796.0000 - val_tn0.3: 15682150.0000 - val_fn0.3: 584020.0000 - val_precision0.3: 0.7597 - val_recall0.3: 0.8154 - val_tp0.5: 2382089.
    0000 - val_fp0.5: 450657.0000 - val_tn0.5: 16047289.0000 - val_fn0.5: 780765.0000 - val_precision0.5: 0.8409 - val_recall0.5: 0.7531 - val_tp0.7: 2118583.0000 - val_fp0.7: 220753.0000 - va
    l_tn0.7: 16277193.0000 - val_fn0.7: 1044271.0000 - val_precision0.7: 0.9056 - val_recall0.7: 0.6698 - val_tp0.9: 1609924.0000 - val_fp0.9: 50267.0000 - val_tn0.9: 16447679.0000 - val_fn0.9
    : 1552930.0000 - val_precision0.9: 0.9697 - val_recall0.9: 0.5090 - val_accuracy: 0.9374 - val_auc: 0.9295 - val_f1: 0.2772
    --- Running training session 59/140
    {'hp_epochs': 20, 'hp_batch_size': 6, 'hp_scaler': 'maxabs', 'hp_n_levels': 3, 'hp_first_filters': 16, 'hp_pool_size': 2, 'hp_input_size': 4096, 'hp_lr_start': 0.06810286704342322, 'hp_lr_
    power': 5.0}
    --- repeat #: 1
    input - shape:   (None, 4096, 1)
    output - shape:  (None, 4096, 1)
    Epoch 1/20
    800/800 [==============================] - 40s 33ms/step - loss: 1.2638 - tp0.1: 2568942.0000 - fp0.1: 8167222.0000 - tn0.1: 8515746.0000 - fn0.1: 408890.0000 - precision0.1: 0.2393 - reca
    ll0.1: 0.8627 - tp0.3: 1525563.0000 - fp0.3: 1857776.0000 - tn0.3: 14825192.0000 - fn0.3: 1452269.0000 - precision0.3: 0.4509 - recall0.3: 0.5123 - tp0.5: 952545.0000 - fp0.5: 543268.0000
    - tn0.5: 16139700.0000 - fn0.5: 2025287.0000 - precision0.5: 0.6368 - recall0.5: 0.3199 - tp0.7: 600606.0000 - fp0.7: 171741.0000 - tn0.7: 16511227.0000 - fn0.7: 2377226.0000 - precision0.
    7: 0.7776 - recall0.7: 0.2017 - tp0.9: 266111.0000 - fp0.9: 28495.0000 - tn0.9: 16654473.0000 - fn0.9: 2711721.0000 - precision0.9: 0.9033 - recall0.9: 0.0894 - accuracy: 0.8694 - auc: 0.7
    995 - f1: 0.2631 - val_loss: 3.2184 - val_tp0.1: 785569.0000 - val_fp0.1: 3481300.0000 - val_tn0.1: 630043.0000 - val_fn0.1: 18288.0000 - val_precision0.1: 0.1841 - val_recall0.1: 0.9772 -
     val_tp0.3: 774801.0000 - val_fp0.3: 3330916.0000 - val_tn0.3: 780427.0000 - val_fn0.3: 29056.0000 - val_precision0.3: 0.1887 - val_recall0.3: 0.9639 - val_tp0.5: 766485.0000 - val_fp0.5:
    3233476.0000 - val_tn0.5: 877867.0000 - val_fn0.5: 37372.0000 - val_precision0.5: 0.1916 - val_recall0.5: 0.9535 - val_tp0.7: 754282.0000 - val_fp0.7: 3093977.0000 - val_tn0.7: 1017366.000
    0 - val_fn0.7: 49575.0000 - val_precision0.7: 0.1960 - val_recall0.7: 0.9383 - val_tp0.9: 704600.0000 - val_fp0.9: 2525773.0000 - val_tn0.9: 1585570.0000 - val_fn0.9: 99257.0000 - val_prec
    ision0.9: 0.2181 - val_recall0.9: 0.8765 - val_accuracy: 0.3345 - val_auc: 0.7196 - val_f1: 0.2811
    Epoch 2/20
    800/800 [==============================] - 23s 28ms/step - loss: 1.0608 - tp0.1: 2627985.0000 - fp0.1: 7509687.0000 - tn0.1: 9173281.0000 - fn0.1: 349847.0000 - precision0.1: 0.2592 - reca
    ll0.1: 0.8825 - tp0.3: 1657396.0000 - fp0.3: 1642042.0000 - tn0.3: 15040926.0000 - fn0.3: 1320436.0000 - precision0.3: 0.5023 - recall0.3: 0.5566 - tp0.5: 1186375.0000 - fp0.5: 597404.0000
     - tn0.5: 16085564.0000 - fn0.5: 1791457.0000 - precision0.5: 0.6651 - recall0.5: 0.3984 - tp0.7: 796587.0000 - fp0.7: 190824.0000 - tn0.7: 16492144.0000 - fn0.7: 2181245.0000 - precision0
    .7: 0.8067 - recall0.7: 0.2675 - tp0.9: 382973.0000 - fp0.9: 30617.0000 - tn0.9: 16652351.0000 - fn0.9: 2594859.0000 - precision0.9: 0.9260 - recall0.9: 0.1286 - accuracy: 0.8785 - auc: 0.
    8268 - f1: 0.2631 - val_loss: 2.8384 - val_tp0.1: 770043.0000 - val_fp0.1: 3359592.0000 - val_tn0.1: 751751.0000 - val_fn0.1: 33814.0000 - val_precision0.1: 0.1865 - val_recall0.1: 0.9579
    - val_tp0.3: 728149.0000 - val_fp0.3: 3009354.0000 - val_tn0.3: 1101989.0000 - val_fn0.3: 75708.0000 - val_precision0.3: 0.1948 - val_recall0.3: 0.9058 - val_tp0.5: 698943.0000 - val_fp0.5
    : 2825281.0000 - val_tn0.5: 1286062.0000 - val_fn0.5: 104914.0000 - val_precision0.5: 0.1983 - val_recall0.5: 0.8695 - val_tp0.7: 650069.0000 - val_fp0.7: 2558191.0000 - val_tn0.7: 1553152
    .0000 - val_fn0.7: 153788.0000 - val_precision0.7: 0.2026 - val_recall0.7: 0.8087 - val_tp0.9: 523649.0000 - val_fp0.9: 1973892.0000 - val_tn0.9: 2137451.0000 - val_fn0.9: 280208.0000 - va
    l_precision0.9: 0.2097 - val_recall0.9: 0.6514 - val_accuracy: 0.4039 - val_auc: 0.6320 - val_f1: 0.2811
    Epoch 3/20
    800/800 [==============================] - 22s 28ms/step - loss: 0.9210 - tp0.1: 2627224.0000 - fp0.1: 6591219.0000 - tn0.1: 10091749.0000 - fn0.1: 350608.0000 - precision0.1: 0.2850 - rec
    all0.1: 0.8823 - tp0.3: 1806105.0000 - fp0.3: 1745398.0000 - tn0.3: 14937570.0000 - fn0.3: 1171727.0000 - precision0.3: 0.5085 - recall0.3: 0.6065 - tp0.5: 1334712.0000 - fp0.5: 661778.000
    0 - tn0.5: 16021190.0000 - fn0.5: 1643120.0000 - precision0.5: 0.6685 - recall0.5: 0.4482 - tp0.7: 898005.0000 - fp0.7: 202721.0000 - tn0.7: 16480247.0000 - fn0.7: 2079827.0000 - precision
    0.7: 0.8158 - recall0.7: 0.3016 - tp0.9: 425030.0000 - fp0.9: 26905.0000 - tn0.9: 16656063.0000 - fn0.9: 2552802.0000 - precision0.9: 0.9405 - recall0.9: 0.1427 - accuracy: 0.8828 - auc: 0
    .8423 - f1: 0.2631 - val_loss: 1.2386 - val_tp0.1: 748635.0000 - val_fp0.1: 2823638.0000 - val_tn0.1: 1287705.0000 - val_fn0.1: 55222.0000 - val_precision0.1: 0.2096 - val_recall0.1: 0.931
    3 - val_tp0.3: 581617.0000 - val_fp0.3: 1541416.0000 - val_tn0.3: 2569927.0000 - val_fn0.3: 222240.0000 - val_precision0.3: 0.2740 - val_recall0.3: 0.7235 - val_tp0.5: 438469.0000 - val_fp
    0.5: 838651.0000 - val_tn0.5: 3272692.0000 - val_fn0.5: 365388.0000 - val_precision0.5: 0.3433 - val_recall0.5: 0.5455 - val_tp0.7: 283136.0000 - val_fp0.7: 250330.0000 - val_tn0.7: 386101
    3.0000 - val_fn0.7: 520721.0000 - val_precision0.7: 0.5307 - val_recall0.7: 0.3522 - val_tp0.9: 100313.0000 - val_fp0.9: 12675.0000 - val_tn0.9: 4098668.0000 - val_fn0.9: 703544.0000 - val
    _precision0.9: 0.8878 - val_recall0.9: 0.1248 - val_accuracy: 0.7550 - val_auc: 0.7546 - val_f1: 0.2811
    Epoch 4/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.8790 - tp0.1: 2622979.0000 - fp0.1: 6082638.0000 - tn0.1: 10600330.0000 - fn0.1: 354853.0000 - precision0.1: 0.3013 - rec
    all0.1: 0.8808 - tp0.3: 1907681.0000 - fp0.3: 1754807.0000 - tn0.3: 14928161.0000 - fn0.3: 1070151.0000 - precision0.3: 0.5209 - recall0.3: 0.6406 - tp0.5: 1430397.0000 - fp0.5: 638842.000
    0 - tn0.5: 16044126.0000 - fn0.5: 1547435.0000 - precision0.5: 0.6913 - recall0.5: 0.4803 - tp0.7: 1005392.0000 - fp0.7: 200266.0000 - tn0.7: 16482702.0000 - fn0.7: 1972440.0000 - precisio
    n0.7: 0.8339 - recall0.7: 0.3376 - tp0.9: 537695.0000 - fp0.9: 32470.0000 - tn0.9: 16650498.0000 - fn0.9: 2440137.0000 - precision0.9: 0.9431 - recall0.9: 0.1806 - accuracy: 0.8888 - auc:
    0.8539 - f1: 0.2631 - val_loss: 1.4382 - val_tp0.1: 759788.0000 - val_fp0.1: 3033777.0000 - val_tn0.1: 1077566.0000 - val_fn0.1: 44069.0000 - val_precision0.1: 0.2003 - val_recall0.1: 0.94
    52 - val_tp0.3: 676790.0000 - val_fp0.3: 2190828.0000 - val_tn0.3: 1920515.0000 - val_fn0.3: 127067.0000 - val_precision0.3: 0.2360 - val_recall0.3: 0.8419 - val_tp0.5: 587381.0000 - val_f
    p0.5: 1457790.0000 - val_tn0.5: 2653553.0000 - val_fn0.5: 216476.0000 - val_precision0.5: 0.2872 - val_recall0.5: 0.7307 - val_tp0.7: 467177.0000 - val_fp0.7: 733730.0000 - val_tn0.7: 3377
    613.0000 - val_fn0.7: 336680.0000 - val_precision0.7: 0.3890 - val_recall0.7: 0.5812 - val_tp0.9: 265920.0000 - val_fp0.9: 128059.0000 - val_tn0.9: 3983284.0000 - val_fn0.9: 537937.0000 -
    val_precision0.9: 0.6750 - val_recall0.9: 0.3308 - val_accuracy: 0.6594 - val_auc: 0.7741 - val_f1: 0.2811
    Epoch 5/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.8323 - tp0.1: 2632941.0000 - fp0.1: 5640241.0000 - tn0.1: 11042727.0000 - fn0.1: 344891.0000 - precision0.1: 0.3183 - rec
    all0.1: 0.8842 - tp0.3: 1987373.0000 - fp0.3: 1723448.0000 - tn0.3: 14959520.0000 - fn0.3: 990459.0000 - precision0.3: 0.5356 - recall0.3: 0.6674 - tp0.5: 1504902.0000 - fp0.5: 660742.0000
     - tn0.5: 16022226.0000 - fn0.5: 1472930.0000 - precision0.5: 0.6949 - recall0.5: 0.5054 - tp0.7: 1062651.0000 - fp0.7: 214795.0000 - tn0.7: 16468173.0000 - fn0.7: 1915181.0000 - precision
    0.7: 0.8319 - recall0.7: 0.3569 - tp0.9: 582194.0000 - fp0.9: 33645.0000 - tn0.9: 16649323.0000 - fn0.9: 2395638.0000 - precision0.9: 0.9454 - recall0.9: 0.1955 - accuracy: 0.8915 - auc: 0
    .8647 - f1: 0.2631 - val_loss: 1.0148 - val_tp0.1: 708531.0000 - val_fp0.1: 1723364.0000 - val_tn0.1: 2387979.0000 - val_fn0.1: 95326.0000 - val_precision0.1: 0.2913 - val_recall0.1: 0.881
    4 - val_tp0.3: 592243.0000 - val_fp0.3: 884014.0000 - val_tn0.3: 3227329.0000 - val_fn0.3: 211614.0000 - val_precision0.3: 0.4012 - val_recall0.3: 0.7368 - val_tp0.5: 463295.0000 - val_fp0
    .5: 407207.0000 - val_tn0.5: 3704136.0000 - val_fn0.5: 340562.0000 - val_precision0.5: 0.5322 - val_recall0.5: 0.5763 - val_tp0.7: 324633.0000 - val_fp0.7: 153907.0000 - val_tn0.7: 3957436
    .0000 - val_fn0.7: 479224.0000 - val_precision0.7: 0.6784 - val_recall0.7: 0.4038 - val_tp0.9: 171994.0000 - val_fp0.9: 27494.0000 - val_tn0.9: 4083849.0000 - val_fn0.9: 631863.0000 - val_
    precision0.9: 0.8622 - val_recall0.9: 0.2140 - val_accuracy: 0.8479 - val_auc: 0.8326 - val_f1: 0.2811
    Epoch 6/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.8051 - tp0.1: 2656801.0000 - fp0.1: 5609482.0000 - tn0.1: 11073486.0000 - fn0.1: 321031.0000 - precision0.1: 0.3214 - rec
    all0.1: 0.8922 - tp0.3: 2004568.0000 - fp0.3: 1721622.0000 - tn0.3: 14961346.0000 - fn0.3: 973264.0000 - precision0.3: 0.5380 - recall0.3: 0.6732 - tp0.5: 1535081.0000 - fp0.5: 636708.0000
     - tn0.5: 16046260.0000 - fn0.5: 1442751.0000 - precision0.5: 0.7068 - recall0.5: 0.5155 - tp0.7: 1104868.0000 - fp0.7: 195727.0000 - tn0.7: 16487241.0000 - fn0.7: 1872964.0000 - precision
    0.7: 0.8495 - recall0.7: 0.3710 - tp0.9: 638669.0000 - fp0.9: 30016.0000 - tn0.9: 16652952.0000 - fn0.9: 2339163.0000 - precision0.9: 0.9551 - recall0.9: 0.2145 - accuracy: 0.8942 - auc: 0
    .8708 - f1: 0.2631 - val_loss: 1.1897 - val_tp0.1: 687813.0000 - val_fp0.1: 1965439.0000 - val_tn0.1: 2145904.0000 - val_fn0.1: 116044.0000 - val_precision0.1: 0.2592 - val_recall0.1: 0.85
    56 - val_tp0.3: 589213.0000 - val_fp0.3: 1346718.0000 - val_tn0.3: 2764625.0000 - val_fn0.3: 214644.0000 - val_precision0.3: 0.3044 - val_recall0.3: 0.7330 - val_tp0.5: 480400.0000 - val_f
    p0.5: 797605.0000 - val_tn0.5: 3313738.0000 - val_fn0.5: 323457.0000 - val_precision0.5: 0.3759 - val_recall0.5: 0.5976 - val_tp0.7: 356338.0000 - val_fp0.7: 349313.0000 - val_tn0.7: 37620
    30.0000 - val_fn0.7: 447519.0000 - val_precision0.7: 0.5050 - val_recall0.7: 0.4433 - val_tp0.9: 203593.0000 - val_fp0.9: 63510.0000 - val_tn0.9: 4047833.0000 - val_fn0.9: 600264.0000 - va
    l_precision0.9: 0.7622 - val_recall0.9: 0.2533 - val_accuracy: 0.7719 - val_auc: 0.7799 - val_f1: 0.2811
    Epoch 7/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.7730 - tp0.1: 2661167.0000 - fp0.1: 5252891.0000 - tn0.1: 11430077.0000 - fn0.1: 316665.0000 - precision0.1: 0.3363 - rec
    all0.1: 0.8937 - tp0.3: 2045078.0000 - fp0.3: 1661108.0000 - tn0.3: 15021860.0000 - fn0.3: 932754.0000 - precision0.3: 0.5518 - recall0.3: 0.6868 - tp0.5: 1572828.0000 - fp0.5: 627634.0000
     - tn0.5: 16055334.0000 - fn0.5: 1405004.0000 - precision0.5: 0.7148 - recall0.5: 0.5282 - tp0.7: 1148247.0000 - fp0.7: 206720.0000 - tn0.7: 16476248.0000 - fn0.7: 1829585.0000 - precision
    0.7: 0.8474 - recall0.7: 0.3856 - tp0.9: 702777.0000 - fp0.9: 37332.0000 - tn0.9: 16645636.0000 - fn0.9: 2275055.0000 - precision0.9: 0.9496 - recall0.9: 0.2360 - accuracy: 0.8966 - auc: 0
    .8779 - f1: 0.2631 - val_loss: 0.9848 - val_tp0.1: 705842.0000 - val_fp0.1: 1730539.0000 - val_tn0.1: 2380804.0000 - val_fn0.1: 98015.0000 - val_precision0.1: 0.2897 - val_recall0.1: 0.878
    1 - val_tp0.3: 495957.0000 - val_fp0.3: 601184.0000 - val_tn0.3: 3510159.0000 - val_fn0.3: 307900.0000 - val_precision0.3: 0.4520 - val_recall0.3: 0.6170 - val_tp0.5: 353962.0000 - val_fp0
    .5: 226006.0000 - val_tn0.5: 3885337.0000 - val_fn0.5: 449895.0000 - val_precision0.5: 0.6103 - val_recall0.5: 0.4403 - val_tp0.7: 232086.0000 - val_fp0.7: 57808.0000 - val_tn0.7: 4053535.
    0000 - val_fn0.7: 571771.0000 - val_precision0.7: 0.8006 - val_recall0.7: 0.2887 - val_tp0.9: 109899.0000 - val_fp0.9: 4377.0000 - val_tn0.9: 4106966.0000 - val_fn0.9: 693958.0000 - val_pr
    ecision0.9: 0.9617 - val_recall0.9: 0.1367 - val_accuracy: 0.8625 - val_auc: 0.8220 - val_f1: 0.2811
    Epoch 8/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.7537 - tp0.1: 2668427.0000 - fp0.1: 5092328.0000 - tn0.1: 11590640.0000 - fn0.1: 309405.0000 - precision0.1: 0.3438 - rec
    all0.1: 0.8961 - tp0.3: 2091292.0000 - fp0.3: 1655195.0000 - tn0.3: 15027773.0000 - fn0.3: 886540.0000 - precision0.3: 0.5582 - recall0.3: 0.7023 - tp0.5: 1591876.0000 - fp0.5: 604215.0000
     - tn0.5: 16078753.0000 - fn0.5: 1385956.0000 - precision0.5: 0.7249 - recall0.5: 0.5346 - tp0.7: 1154686.0000 - fp0.7: 198461.0000 - tn0.7: 16484507.0000 - fn0.7: 1823146.0000 - precision
    0.7: 0.8533 - recall0.7: 0.3878 - tp0.9: 706986.0000 - fp0.9: 36859.0000 - tn0.9: 16646109.0000 - fn0.9: 2270846.0000 - precision0.9: 0.9504 - recall0.9: 0.2374 - accuracy: 0.8988 - auc: 0
    .8827 - f1: 0.2631 - val_loss: 1.0536 - val_tp0.1: 717248.0000 - val_fp0.1: 2127277.0000 - val_tn0.1: 1984066.0000 - val_fn0.1: 86609.0000 - val_precision0.1: 0.2522 - val_recall0.1: 0.892
    3 - val_tp0.3: 516820.0000 - val_fp0.3: 718092.0000 - val_tn0.3: 3393251.0000 - val_fn0.3: 287037.0000 - val_precision0.3: 0.4185 - val_recall0.3: 0.6429 - val_tp0.5: 349749.0000 - val_fp0
    .5: 209234.0000 - val_tn0.5: 3902109.0000 - val_fn0.5: 454108.0000 - val_precision0.5: 0.6257 - val_recall0.5: 0.4351 - val_tp0.7: 217539.0000 - val_fp0.7: 55114.0000 - val_tn0.7: 4056229.
    0000 - val_fn0.7: 586318.0000 - val_precision0.7: 0.7979 - val_recall0.7: 0.2706 - val_tp0.9: 108545.0000 - val_fp0.9: 6637.0000 - val_tn0.9: 4104706.0000 - val_fn0.9: 695312.0000 - val_pr
    ecision0.9: 0.9424 - val_recall0.9: 0.1350 - val_accuracy: 0.8650 - val_auc: 0.8110 - val_f1: 0.2811
    Epoch 9/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.7245 - tp0.1: 2670184.0000 - fp0.1: 4736653.0000 - tn0.1: 11946315.0000 - fn0.1: 307648.0000 - precision0.1: 0.3605 - rec
    all0.1: 0.8967 - tp0.3: 2138051.0000 - fp0.3: 1655069.0000 - tn0.3: 15027899.0000 - fn0.3: 839781.0000 - precision0.3: 0.5637 - recall0.3: 0.7180 - tp0.5: 1658889.0000 - fp0.5: 602716.0000
     - tn0.5: 16080252.0000 - fn0.5: 1318943.0000 - precision0.5: 0.7335 - recall0.5: 0.5571 - tp0.7: 1231743.0000 - fp0.7: 199604.0000 - tn0.7: 16483364.0000 - fn0.7: 1746089.0000 - precision
    0.7: 0.8605 - recall0.7: 0.4136 - tp0.9: 770284.0000 - fp0.9: 38798.0000 - tn0.9: 16644170.0000 - fn0.9: 2207548.0000 - precision0.9: 0.9520 - recall0.9: 0.2587 - accuracy: 0.9023 - auc: 0
    .8890 - f1: 0.2631 - val_loss: 0.9833 - val_tp0.1: 684923.0000 - val_fp0.1: 1482632.0000 - val_tn0.1: 2628711.0000 - val_fn0.1: 118934.0000 - val_precision0.1: 0.3160 - val_recall0.1: 0.85
    20 - val_tp0.3: 378339.0000 - val_fp0.3: 205771.0000 - val_tn0.3: 3905572.0000 - val_fn0.3: 425518.0000 - val_precision0.3: 0.6477 - val_recall0.3: 0.4707 - val_tp0.5: 214088.0000 - val_fp
    0.5: 39231.0000 - val_tn0.5: 4072112.0000 - val_fn0.5: 589769.0000 - val_precision0.5: 0.8451 - val_recall0.5: 0.2663 - val_tp0.7: 124578.0000 - val_fp0.7: 7623.0000 - val_tn0.7: 4103720.0
    000 - val_fn0.7: 679279.0000 - val_precision0.7: 0.9423 - val_recall0.7: 0.1550 - val_tp0.9: 59851.0000 - val_fp0.9: 608.0000 - val_tn0.9: 4110735.0000 - val_fn0.9: 744006.0000 - val_preci
    sion0.9: 0.9899 - val_recall0.9: 0.0745 - val_accuracy: 0.8720 - val_auc: 0.8270 - val_f1: 0.2811
    Epoch 10/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.7278 - tp0.1: 2665983.0000 - fp0.1: 4770602.0000 - tn0.1: 11912366.0000 - fn0.1: 311849.0000 - precision0.1: 0.3585 - rec
    all0.1: 0.8953 - tp0.3: 2137054.0000 - fp0.3: 1618394.0000 - tn0.3: 15064574.0000 - fn0.3: 840778.0000 - precision0.3: 0.5691 - recall0.3: 0.7177 - tp0.5: 1657865.0000 - fp0.5: 592475.0000
     - tn0.5: 16090493.0000 - fn0.5: 1319967.0000 - precision0.5: 0.7367 - recall0.5: 0.5567 - tp0.7: 1234306.0000 - fp0.7: 193079.0000 - tn0.7: 16489889.0000 - fn0.7: 1743526.0000 - precision
    0.7: 0.8647 - recall0.7: 0.4145 - tp0.9: 773361.0000 - fp0.9: 37289.0000 - tn0.9: 16645679.0000 - fn0.9: 2204471.0000 - precision0.9: 0.9540 - recall0.9: 0.2597 - accuracy: 0.9027 - auc: 0
    .8886 - f1: 0.2631 - val_loss: 0.9507 - val_tp0.1: 693260.0000 - val_fp0.1: 1463538.0000 - val_tn0.1: 2647805.0000 - val_fn0.1: 110597.0000 - val_precision0.1: 0.3214 - val_recall0.1: 0.86
    24 - val_tp0.3: 398366.0000 - val_fp0.3: 212804.0000 - val_tn0.3: 3898539.0000 - val_fn0.3: 405491.0000 - val_precision0.3: 0.6518 - val_recall0.3: 0.4956 - val_tp0.5: 241139.0000 - val_fp
    0.5: 41379.0000 - val_tn0.5: 4069964.0000 - val_fn0.5: 562718.0000 - val_precision0.5: 0.8535 - val_recall0.5: 0.3000 - val_tp0.7: 143798.0000 - val_fp0.7: 8441.0000 - val_tn0.7: 4102902.0
    000 - val_fn0.7: 660059.0000 - val_precision0.7: 0.9446 - val_recall0.7: 0.1789 - val_tp0.9: 73993.0000 - val_fp0.9: 945.0000 - val_tn0.9: 4110398.0000 - val_fn0.9: 729864.0000 - val_preci
    sion0.9: 0.9874 - val_recall0.9: 0.0920 - val_accuracy: 0.8771 - val_auc: 0.8421 - val_f1: 0.2811
    Epoch 11/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.7098 - tp0.1: 2671173.0000 - fp0.1: 4583226.0000 - tn0.1: 12099742.0000 - fn0.1: 306659.0000 - precision0.1: 0.3682 - rec
    all0.1: 0.8970 - tp0.3: 2161831.0000 - fp0.3: 1611405.0000 - tn0.3: 15071563.0000 - fn0.3: 816001.0000 - precision0.3: 0.5729 - recall0.3: 0.7260 - tp0.5: 1683781.0000 - fp0.5: 593951.0000
     - tn0.5: 16089017.0000 - fn0.5: 1294051.0000 - precision0.5: 0.7392 - recall0.5: 0.5654 - tp0.7: 1246509.0000 - fp0.7: 197501.0000 - tn0.7: 16485467.0000 - fn0.7: 1731323.0000 - precision
    0.7: 0.8632 - recall0.7: 0.4186 - tp0.9: 780021.0000 - fp0.9: 37187.0000 - tn0.9: 16645781.0000 - fn0.9: 2197811.0000 - precision0.9: 0.9545 - recall0.9: 0.2619 - accuracy: 0.9040 - auc: 0
    .8921 - f1: 0.2631 - val_loss: 0.9881 - val_tp0.1: 684864.0000 - val_fp0.1: 1514096.0000 - val_tn0.1: 2597247.0000 - val_fn0.1: 118993.0000 - val_precision0.1: 0.3114 - val_recall0.1: 0.85
    20 - val_tp0.3: 448578.0000 - val_fp0.3: 437347.0000 - val_tn0.3: 3673996.0000 - val_fn0.3: 355279.0000 - val_precision0.3: 0.5063 - val_recall0.3: 0.5580 - val_tp0.5: 306480.0000 - val_fp
    0.5: 142586.0000 - val_tn0.5: 3968757.0000 - val_fn0.5: 497377.0000 - val_precision0.5: 0.6825 - val_recall0.5: 0.3813 - val_tp0.7: 196534.0000 - val_fp0.7: 34758.0000 - val_tn0.7: 4076585
    .0000 - val_fn0.7: 607323.0000 - val_precision0.7: 0.8497 - val_recall0.7: 0.2445 - val_tp0.9: 93543.0000 - val_fp0.9: 2618.0000 - val_tn0.9: 4108725.0000 - val_fn0.9: 710314.0000 - val_pr
    ecision0.9: 0.9728 - val_recall0.9: 0.1164 - val_accuracy: 0.8698 - val_auc: 0.8226 - val_f1: 0.2811
    Epoch 12/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.7115 - tp0.1: 2672752.0000 - fp0.1: 4630371.0000 - tn0.1: 12052597.0000 - fn0.1: 305080.0000 - precision0.1: 0.3660 - rec
    all0.1: 0.8975 - tp0.3: 2159193.0000 - fp0.3: 1616151.0000 - tn0.3: 15066817.0000 - fn0.3: 818639.0000 - precision0.3: 0.5719 - recall0.3: 0.7251 - tp0.5: 1689796.0000 - fp0.5: 588788.0000
     - tn0.5: 16094180.0000 - fn0.5: 1288036.0000 - precision0.5: 0.7416 - recall0.5: 0.5675 - tp0.7: 1257203.0000 - fp0.7: 187802.0000 - tn0.7: 16495166.0000 - fn0.7: 1720629.0000 - precision
    0.7: 0.8700 - recall0.7: 0.4222 - tp0.9: 781350.0000 - fp0.9: 35537.0000 - tn0.9: 16647431.0000 - fn0.9: 2196482.0000 - precision0.9: 0.9565 - recall0.9: 0.2624 - accuracy: 0.9045 - auc: 0
    .8920 - f1: 0.2631 - val_loss: 0.9426 - val_tp0.1: 694538.0000 - val_fp0.1: 1496227.0000 - val_tn0.1: 2615116.0000 - val_fn0.1: 109319.0000 - val_precision0.1: 0.3170 - val_recall0.1: 0.86
    40 - val_tp0.3: 506023.0000 - val_fp0.3: 520642.0000 - val_tn0.3: 3590701.0000 - val_fn0.3: 297834.0000 - val_precision0.3: 0.4929 - val_recall0.3: 0.6295 - val_tp0.5: 346422.0000 - val_fp
    0.5: 153015.0000 - val_tn0.5: 3958328.0000 - val_fn0.5: 457435.0000 - val_precision0.5: 0.6936 - val_recall0.5: 0.4309 - val_tp0.7: 221842.0000 - val_fp0.7: 37879.0000 - val_tn0.7: 4073464
    .0000 - val_fn0.7: 582015.0000 - val_precision0.7: 0.8542 - val_recall0.7: 0.2760 - val_tp0.9: 104244.0000 - val_fp0.9: 3502.0000 - val_tn0.9: 4107841.0000 - val_fn0.9: 699613.0000 - val_p
    recision0.9: 0.9675 - val_recall0.9: 0.1297 - val_accuracy: 0.8758 - val_auc: 0.8365 - val_f1: 0.2811
    Epoch 13/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.7049 - tp0.1: 2669030.0000 - fp0.1: 4503237.0000 - tn0.1: 12179731.0000 - fn0.1: 308802.0000 - precision0.1: 0.3721 - rec
    all0.1: 0.8963 - tp0.3: 2190933.0000 - fp0.3: 1671712.0000 - tn0.3: 15011256.0000 - fn0.3: 786899.0000 - precision0.3: 0.5672 - recall0.3: 0.7357 - tp0.5: 1721545.0000 - fp0.5: 613878.0000
     - tn0.5: 16069090.0000 - fn0.5: 1256287.0000 - precision0.5: 0.7371 - recall0.5: 0.5781 - tp0.7: 1280739.0000 - fp0.7: 198223.0000 - tn0.7: 16484745.0000 - fn0.7: 1697093.0000 - precision
    0.7: 0.8660 - recall0.7: 0.4301 - tp0.9: 795579.0000 - fp0.9: 35764.0000 - tn0.9: 16647204.0000 - fn0.9: 2182253.0000 - precision0.9: 0.9570 - recall0.9: 0.2672 - accuracy: 0.9049 - auc: 0
    .8932 - f1: 0.2631 - val_loss: 0.9456 - val_tp0.1: 699033.0000 - val_fp0.1: 1563859.0000 - val_tn0.1: 2547484.0000 - val_fn0.1: 104824.0000 - val_precision0.1: 0.3089 - val_recall0.1: 0.86
    96 - val_tp0.3: 540655.0000 - val_fp0.3: 610467.0000 - val_tn0.3: 3500876.0000 - val_fn0.3: 263202.0000 - val_precision0.3: 0.4697 - val_recall0.3: 0.6726 - val_tp0.5: 379380.0000 - val_fp
    0.5: 190187.0000 - val_tn0.5: 3921156.0000 - val_fn0.5: 424477.0000 - val_precision0.5: 0.6661 - val_recall0.5: 0.4719 - val_tp0.7: 250875.0000 - val_fp0.7: 56341.0000 - val_tn0.7: 4055002
    .0000 - val_fn0.7: 552982.0000 - val_precision0.7: 0.8166 - val_recall0.7: 0.3121 - val_tp0.9: 123390.0000 - val_fp0.9: 6935.0000 - val_tn0.9: 4104408.0000 - val_fn0.9: 680467.0000 - val_p
    recision0.9: 0.9468 - val_recall0.9: 0.1535 - val_accuracy: 0.8749 - val_auc: 0.8387 - val_f1: 0.2811
    Epoch 14/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.7025 - tp0.1: 2675299.0000 - fp0.1: 4563270.0000 - tn0.1: 12119698.0000 - fn0.1: 302533.0000 - precision0.1: 0.3696 - rec
    all0.1: 0.8984 - tp0.3: 2177828.0000 - fp0.3: 1603216.0000 - tn0.3: 15079752.0000 - fn0.3: 800004.0000 - precision0.3: 0.5760 - recall0.3: 0.7313 - tp0.5: 1706323.0000 - fp0.5: 592574.0000
     - tn0.5: 16090394.0000 - fn0.5: 1271509.0000 - precision0.5: 0.7422 - recall0.5: 0.5730 - tp0.7: 1267052.0000 - fp0.7: 191517.0000 - tn0.7: 16491451.0000 - fn0.7: 1710780.0000 - precision
    0.7: 0.8687 - recall0.7: 0.4255 - tp0.9: 779803.0000 - fp0.9: 33525.0000 - tn0.9: 16649443.0000 - fn0.9: 2198029.0000 - precision0.9: 0.9588 - recall0.9: 0.2619 - accuracy: 0.9052 - auc: 0
    .8942 - f1: 0.2631 - val_loss: 0.9513 - val_tp0.1: 685976.0000 - val_fp0.1: 1445496.0000 - val_tn0.1: 2665847.0000 - val_fn0.1: 117881.0000 - val_precision0.1: 0.3218 - val_recall0.1: 0.85
    34 - val_tp0.3: 502057.0000 - val_fp0.3: 484827.0000 - val_tn0.3: 3626516.0000 - val_fn0.3: 301800.0000 - val_precision0.3: 0.5087 - val_recall0.3: 0.6246 - val_tp0.5: 350620.0000 - val_fp
    0.5: 151636.0000 - val_tn0.5: 3959707.0000 - val_fn0.5: 453237.0000 - val_precision0.5: 0.6981 - val_recall0.5: 0.4362 - val_tp0.7: 231119.0000 - val_fp0.7: 44841.0000 - val_tn0.7: 4066502
    .0000 - val_fn0.7: 572738.0000 - val_precision0.7: 0.8375 - val_recall0.7: 0.2875 - val_tp0.9: 117775.0000 - val_fp0.9: 6325.0000 - val_tn0.9: 4105018.0000 - val_fn0.9: 686082.0000 - val_p
    recision0.9: 0.9490 - val_recall0.9: 0.1465 - val_accuracy: 0.8769 - val_auc: 0.8355 - val_f1: 0.2811
    Epoch 15/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.6970 - tp0.1: 2674241.0000 - fp0.1: 4479713.0000 - tn0.1: 12203255.0000 - fn0.1: 303591.0000 - precision0.1: 0.3738 - rec
    all0.1: 0.8980 - tp0.3: 2170500.0000 - fp0.3: 1588775.0000 - tn0.3: 15094193.0000 - fn0.3: 807332.0000 - precision0.3: 0.5774 - recall0.3: 0.7289 - tp0.5: 1716769.0000 - fp0.5: 596411.0000
     - tn0.5: 16086557.0000 - fn0.5: 1261063.0000 - precision0.5: 0.7422 - recall0.5: 0.5765 - tp0.7: 1295621.0000 - fp0.7: 194669.0000 - tn0.7: 16488299.0000 - fn0.7: 1682211.0000 - precision
    0.7: 0.8694 - recall0.7: 0.4351 - tp0.9: 814054.0000 - fp0.9: 35899.0000 - tn0.9: 16647069.0000 - fn0.9: 2163778.0000 - precision0.9: 0.9578 - recall0.9: 0.2734 - accuracy: 0.9055 - auc: 0
    .8945 - f1: 0.2631 - val_loss: 0.9517 - val_tp0.1: 688515.0000 - val_fp0.1: 1481139.0000 - val_tn0.1: 2630204.0000 - val_fn0.1: 115342.0000 - val_precision0.1: 0.3173 - val_recall0.1: 0.85
    65 - val_tp0.3: 506409.0000 - val_fp0.3: 477938.0000 - val_tn0.3: 3633405.0000 - val_fn0.3: 297448.0000 - val_precision0.3: 0.5145 - val_recall0.3: 0.6300 - val_tp0.5: 352279.0000 - val_fp
    0.5: 150937.0000 - val_tn0.5: 3960406.0000 - val_fn0.5: 451578.0000 - val_precision0.5: 0.7001 - val_recall0.5: 0.4382 - val_tp0.7: 231093.0000 - val_fp0.7: 45551.0000 - val_tn0.7: 4065792
    .0000 - val_fn0.7: 572764.0000 - val_precision0.7: 0.8353 - val_recall0.7: 0.2875 - val_tp0.9: 118279.0000 - val_fp0.9: 6604.0000 - val_tn0.9: 4104739.0000 - val_fn0.9: 685578.0000 - val_p
    recision0.9: 0.9471 - val_recall0.9: 0.1471 - val_accuracy: 0.8774 - val_auc: 0.8370 - val_f1: 0.2811
    Epoch 16/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.6972 - tp0.1: 2674892.0000 - fp0.1: 4478986.0000 - tn0.1: 12203982.0000 - fn0.1: 302940.0000 - precision0.1: 0.3739 - rec
    all0.1: 0.8983 - tp0.3: 2177848.0000 - fp0.3: 1577462.0000 - tn0.3: 15105506.0000 - fn0.3: 799984.0000 - precision0.3: 0.5799 - recall0.3: 0.7314 - tp0.5: 1710580.0000 - fp0.5: 588844.0000
     - tn0.5: 16094124.0000 - fn0.5: 1267252.0000 - precision0.5: 0.7439 - recall0.5: 0.5744 - tp0.7: 1276251.0000 - fp0.7: 191455.0000 - tn0.7: 16491513.0000 - fn0.7: 1701581.0000 - precision
    0.7: 0.8696 - recall0.7: 0.4286 - tp0.9: 794990.0000 - fp0.9: 33049.0000 - tn0.9: 16649919.0000 - fn0.9: 2182842.0000 - precision0.9: 0.9601 - recall0.9: 0.2670 - accuracy: 0.9056 - auc: 0
    .8951 - f1: 0.2631 - val_loss: 0.9505 - val_tp0.1: 684484.0000 - val_fp0.1: 1423827.0000 - val_tn0.1: 2687516.0000 - val_fn0.1: 119373.0000 - val_precision0.1: 0.3247 - val_recall0.1: 0.85
    15 - val_tp0.3: 507858.0000 - val_fp0.3: 511252.0000 - val_tn0.3: 3600091.0000 - val_fn0.3: 295999.0000 - val_precision0.3: 0.4983 - val_recall0.3: 0.6318 - val_tp0.5: 356641.0000 - val_fp
    0.5: 160219.0000 - val_tn0.5: 3951124.0000 - val_fn0.5: 447216.0000 - val_precision0.5: 0.6900 - val_recall0.5: 0.4437 - val_tp0.7: 236723.0000 - val_fp0.7: 47470.0000 - val_tn0.7: 4063873
    .0000 - val_fn0.7: 567134.0000 - val_precision0.7: 0.8330 - val_recall0.7: 0.2945 - val_tp0.9: 119706.0000 - val_fp0.9: 6487.0000 - val_tn0.9: 4104856.0000 - val_fn0.9: 684151.0000 - val_p
    recision0.9: 0.9486 - val_recall0.9: 0.1489 - val_accuracy: 0.8764 - val_auc: 0.8350 - val_f1: 0.2811
    Epoch 17/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.6981 - tp0.1: 2674601.0000 - fp0.1: 4489171.0000 - tn0.1: 12193797.0000 - fn0.1: 303231.0000 - precision0.1: 0.3734 - rec
    all0.1: 0.8982 - tp0.3: 2183675.0000 - fp0.3: 1573762.0000 - tn0.3: 15109206.0000 - fn0.3: 794157.0000 - precision0.3: 0.5812 - recall0.3: 0.7333 - tp0.5: 1713199.0000 - fp0.5: 586718.0000
     - tn0.5: 16096250.0000 - fn0.5: 1264633.0000 - precision0.5: 0.7449 - recall0.5: 0.5753 - tp0.7: 1283634.0000 - fp0.7: 195576.0000 - tn0.7: 16487392.0000 - fn0.7: 1694198.0000 - precision
    0.7: 0.8678 - recall0.7: 0.4311 - tp0.9: 805115.0000 - fp0.9: 36004.0000 - tn0.9: 16646964.0000 - fn0.9: 2172717.0000 - precision0.9: 0.9572 - recall0.9: 0.2704 - accuracy: 0.9058 - auc: 0
    .8950 - f1: 0.2631 - val_loss: 0.9465 - val_tp0.1: 687031.0000 - val_fp0.1: 1439264.0000 - val_tn0.1: 2672079.0000 - val_fn0.1: 116826.0000 - val_precision0.1: 0.3231 - val_recall0.1: 0.85
    47 - val_tp0.3: 508441.0000 - val_fp0.3: 510821.0000 - val_tn0.3: 3600522.0000 - val_fn0.3: 295416.0000 - val_precision0.3: 0.4988 - val_recall0.3: 0.6325 - val_tp0.5: 352823.0000 - val_fp
    0.5: 152954.0000 - val_tn0.5: 3958389.0000 - val_fn0.5: 451034.0000 - val_precision0.5: 0.6976 - val_recall0.5: 0.4389 - val_tp0.7: 231208.0000 - val_fp0.7: 43355.0000 - val_tn0.7: 4067988
    .0000 - val_fn0.7: 572649.0000 - val_precision0.7: 0.8421 - val_recall0.7: 0.2876 - val_tp0.9: 114713.0000 - val_fp0.9: 5450.0000 - val_tn0.9: 4105893.0000 - val_fn0.9: 689144.0000 - val_p
    recision0.9: 0.9546 - val_recall0.9: 0.1427 - val_accuracy: 0.8771 - val_auc: 0.8361 - val_f1: 0.2811
    Epoch 18/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.6969 - tp0.1: 2676617.0000 - fp0.1: 4470158.0000 - tn0.1: 12212810.0000 - fn0.1: 301215.0000 - precision0.1: 0.3745 - rec
    all0.1: 0.8988 - tp0.3: 2177086.0000 - fp0.3: 1601828.0000 - tn0.3: 15081140.0000 - fn0.3: 800746.0000 - precision0.3: 0.5761 - recall0.3: 0.7311 - tp0.5: 1715470.0000 - fp0.5: 611086.0000
     - tn0.5: 16071882.0000 - fn0.5: 1262362.0000 - precision0.5: 0.7373 - recall0.5: 0.5761 - tp0.7: 1288609.0000 - fp0.7: 198192.0000 - tn0.7: 16484776.0000 - fn0.7: 1689223.0000 - precision
    0.7: 0.8667 - recall0.7: 0.4327 - tp0.9: 807642.0000 - fp0.9: 37272.0000 - tn0.9: 16645696.0000 - fn0.9: 2170190.0000 - precision0.9: 0.9559 - recall0.9: 0.2712 - accuracy: 0.9047 - auc: 0
    .8948 - f1: 0.2631 - val_loss: 0.9503 - val_tp0.1: 684845.0000 - val_fp0.1: 1434583.0000 - val_tn0.1: 2676760.0000 - val_fn0.1: 119012.0000 - val_precision0.1: 0.3231 - val_recall0.1: 0.85
    19 - val_tp0.3: 501142.0000 - val_fp0.3: 471764.0000 - val_tn0.3: 3639579.0000 - val_fn0.3: 302715.0000 - val_precision0.3: 0.5151 - val_recall0.3: 0.6234 - val_tp0.5: 347005.0000 - val_fp
    0.5: 143102.0000 - val_tn0.5: 3968241.0000 - val_fn0.5: 456852.0000 - val_precision0.5: 0.7080 - val_recall0.5: 0.4317 - val_tp0.7: 226199.0000 - val_fp0.7: 40947.0000 - val_tn0.7: 4070396
    .0000 - val_fn0.7: 577658.0000 - val_precision0.7: 0.8467 - val_recall0.7: 0.2814 - val_tp0.9: 113524.0000 - val_fp0.9: 5443.0000 - val_tn0.9: 4105900.0000 - val_fn0.9: 690333.0000 - val_p
    recision0.9: 0.9542 - val_recall0.9: 0.1412 - val_accuracy: 0.8779 - val_auc: 0.8363 - val_f1: 0.2811
    Epoch 19/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.6974 - tp0.1: 2678075.0000 - fp0.1: 4501081.0000 - tn0.1: 12181887.0000 - fn0.1: 299757.0000 - precision0.1: 0.3730 - rec
    all0.1: 0.8993 - tp0.3: 2193569.0000 - fp0.3: 1620047.0000 - tn0.3: 15062921.0000 - fn0.3: 784263.0000 - precision0.3: 0.5752 - recall0.3: 0.7366 - tp0.5: 1721203.0000 - fp0.5: 604162.0000
     - tn0.5: 16078806.0000 - fn0.5: 1256629.0000 - precision0.5: 0.7402 - recall0.5: 0.5780 - tp0.7: 1284180.0000 - fp0.7: 199088.0000 - tn0.7: 16483880.0000 - fn0.7: 1693652.0000 - precision
    0.7: 0.8658 - recall0.7: 0.4312 - tp0.9: 797069.0000 - fp0.9: 36223.0000 - tn0.9: 16646745.0000 - fn0.9: 2180763.0000 - precision0.9: 0.9565 - recall0.9: 0.2677 - accuracy: 0.9054 - auc: 0
    .8954 - f1: 0.2631 - val_loss: 0.9439 - val_tp0.1: 694044.0000 - val_fp0.1: 1504048.0000 - val_tn0.1: 2607295.0000 - val_fn0.1: 109813.0000 - val_precision0.1: 0.3157 - val_recall0.1: 0.86
    34 - val_tp0.3: 523881.0000 - val_fp0.3: 545389.0000 - val_tn0.3: 3565954.0000 - val_fn0.3: 279976.0000 - val_precision0.3: 0.4899 - val_recall0.3: 0.6517 - val_tp0.5: 369154.0000 - val_fp
    0.5: 174373.0000 - val_tn0.5: 3936970.0000 - val_fn0.5: 434703.0000 - val_precision0.5: 0.6792 - val_recall0.5: 0.4592 - val_tp0.7: 245997.0000 - val_fp0.7: 52312.0000 - val_tn0.7: 4059031
    .0000 - val_fn0.7: 557860.0000 - val_precision0.7: 0.8246 - val_recall0.7: 0.3060 - val_tp0.9: 124029.0000 - val_fp0.9: 7043.0000 - val_tn0.9: 4104300.0000 - val_fn0.9: 679828.0000 - val_p
    recision0.9: 0.9463 - val_recall0.9: 0.1543 - val_accuracy: 0.8761 - val_auc: 0.8387 - val_f1: 0.2811
    Epoch 20/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.7034 - tp0.1: 2668995.0000 - fp0.1: 4472643.0000 - tn0.1: 12210325.0000 - fn0.1: 308837.0000 - precision0.1: 0.3737 - rec
    all0.1: 0.8963 - tp0.3: 2164194.0000 - fp0.3: 1572121.0000 - tn0.3: 15110847.0000 - fn0.3: 813638.0000 - precision0.3: 0.5792 - recall0.3: 0.7268 - tp0.5: 1680381.0000 - fp0.5: 579083.0000
     - tn0.5: 16103885.0000 - fn0.5: 1297451.0000 - precision0.5: 0.7437 - recall0.5: 0.5643 - tp0.7: 1241011.0000 - fp0.7: 190475.0000 - tn0.7: 16492493.0000 - fn0.7: 1736821.0000 - precision
    0.7: 0.8669 - recall0.7: 0.4167 - tp0.9: 785981.0000 - fp0.9: 34923.0000 - tn0.9: 16648045.0000 - fn0.9: 2191851.0000 - precision0.9: 0.9575 - recall0.9: 0.2639 - accuracy: 0.9046 - auc: 0
    .8932 - f1: 0.2631 - val_loss: 0.9520 - val_tp0.1: 682980.0000 - val_fp0.1: 1413218.0000 - val_tn0.1: 2698125.0000 - val_fn0.1: 120877.0000 - val_precision0.1: 0.3258 - val_recall0.1: 0.84
    96 - val_tp0.3: 500974.0000 - val_fp0.3: 498780.0000 - val_tn0.3: 3612563.0000 - val_fn0.3: 302883.0000 - val_precision0.3: 0.5011 - val_recall0.3: 0.6232 - val_tp0.5: 348018.0000 - val_fp
    0.5: 150788.0000 - val_tn0.5: 3960555.0000 - val_fn0.5: 455839.0000 - val_precision0.5: 0.6977 - val_recall0.5: 0.4329 - val_tp0.7: 228527.0000 - val_fp0.7: 42760.0000 - val_tn0.7: 4068583
    .0000 - val_fn0.7: 575330.0000 - val_precision0.7: 0.8424 - val_recall0.7: 0.2843 - val_tp0.9: 113851.0000 - val_fp0.9: 5410.0000 - val_tn0.9: 4105933.0000 - val_fn0.9: 690006.0000 - val_p
    recision0.9: 0.9546 - val_recall0.9: 0.1416 - val_accuracy: 0.8766 - val_auc: 0.8340 - val_f1: 0.2811
    --- Running training session 60/140
    {'hp_epochs': 20, 'hp_batch_size': 6, 'hp_scaler': 'maxabs', 'hp_n_levels': 3, 'hp_first_filters': 16, 'hp_pool_size': 2, 'hp_input_size': 4096, 'hp_lr_start': 0.06810286704342322, 'hp_lr_
    power': 5.0}
    --- repeat #: 2
    input - shape:   (None, 4096, 1)
    output - shape:  (None, 4096, 1)
    Epoch 1/20
    800/800 [==============================] - 41s 33ms/step - loss: 1.1728 - tp0.1: 2643619.0000 - fp0.1: 9398198.0000 - tn0.1: 7284770.0000 - fn0.1: 334213.0000 - precision0.1: 0.2195 - reca
    ll0.1: 0.8878 - tp0.3: 1267730.0000 - fp0.3: 3106478.0000 - tn0.3: 13576490.0000 - fn0.3: 1710102.0000 - precision0.3: 0.2898 - recall0.3: 0.4257 - tp0.5: 110715.0000 - fp0.5: 152305.0000
    - tn0.5: 16530663.0000 - fn0.5: 2867117.0000 - precision0.5: 0.4209 - recall0.5: 0.0372 - tp0.7: 22534.0000 - fp0.7: 27016.0000 - tn0.7: 16655952.0000 - fn0.7: 2955298.0000 - precision0.7:
     0.4548 - recall0.7: 0.0076 - tp0.9: 5370.0000 - fp0.9: 4276.0000 - tn0.9: 16678692.0000 - fn0.9: 2972462.0000 - precision0.9: 0.5567 - recall0.9: 0.0018 - accuracy: 0.8464 - auc: 0.7210 -
     f1: 0.2631 - val_loss: 1.3911 - val_tp0.1: 803682.0000 - val_fp0.1: 4109861.0000 - val_tn0.1: 1482.0000 - val_fn0.1: 175.0000 - val_precision0.1: 0.1636 - val_recall0.1: 0.9998 - val_tp0.
    3: 481990.0000 - val_fp0.3: 1381226.0000 - val_tn0.3: 2730117.0000 - val_fn0.3: 321867.0000 - val_precision0.3: 0.2587 - val_recall0.3: 0.5996 - val_tp0.5: 173457.0000 - val_fp0.5: 387503.
    0000 - val_tn0.5: 3723840.0000 - val_fn0.5: 630400.0000 - val_precision0.5: 0.3092 - val_recall0.5: 0.2158 - val_tp0.7: 62452.0000 - val_fp0.7: 51611.0000 - val_tn0.7: 4059732.0000 - val_f
    n0.7: 741405.0000 - val_precision0.7: 0.5475 - val_recall0.7: 0.0777 - val_tp0.9: 36.0000 - val_fp0.9: 0.0000e+00 - val_tn0.9: 4111343.0000 - val_fn0.9: 803821.0000 - val_precision0.9: 1.0
    000 - val_recall0.9: 4.4784e-05 - val_accuracy: 0.7929 - val_auc: 0.6622 - val_f1: 0.2811
    Epoch 2/20
    800/800 [==============================] - 23s 28ms/step - loss: 1.0815 - tp0.1: 2577878.0000 - fp0.1: 8005433.0000 - tn0.1: 8677535.0000 - fn0.1: 399954.0000 - precision0.1: 0.2436 - reca
    ll0.1: 0.8657 - tp0.3: 1447126.0000 - fp0.3: 2241046.0000 - tn0.3: 14441922.0000 - fn0.3: 1530706.0000 - precision0.3: 0.3924 - recall0.3: 0.4860 - tp0.5: 687354.0000 - fp0.5: 398807.0000
    - tn0.5: 16284161.0000 - fn0.5: 2290478.0000 - precision0.5: 0.6328 - recall0.5: 0.2308 - tp0.7: 413817.0000 - fp0.7: 98920.0000 - tn0.7: 16584048.0000 - fn0.7: 2564015.0000 - precision0.7
    : 0.8071 - recall0.7: 0.1390 - tp0.9: 186771.0000 - fp0.9: 14141.0000 - tn0.9: 16668827.0000 - fn0.9: 2791061.0000 - precision0.9: 0.9296 - recall0.9: 0.0627 - accuracy: 0.8632 - auc: 0.77
    95 - f1: 0.2631 - val_loss: 2.8986 - val_tp0.1: 803131.0000 - val_fp0.1: 4099762.0000 - val_tn0.1: 11581.0000 - val_fn0.1: 726.0000 - val_precision0.1: 0.1638 - val_recall0.1: 0.9991 - val
    _tp0.3: 802096.0000 - val_fp0.3: 4084853.0000 - val_tn0.3: 26490.0000 - val_fn0.3: 1761.0000 - val_precision0.3: 0.1641 - val_recall0.3: 0.9978 - val_tp0.5: 651539.0000 - val_fp0.5: 192398
    5.0000 - val_tn0.5: 2187358.0000 - val_fn0.5: 152318.0000 - val_precision0.5: 0.2530 - val_recall0.5: 0.8105 - val_tp0.7: 625381.0000 - val_fp0.7: 1805820.0000 - val_tn0.7: 2305523.0000 -
    val_fn0.7: 178476.0000 - val_precision0.7: 0.2572 - val_recall0.7: 0.7780 - val_tp0.9: 585910.0000 - val_fp0.9: 1628567.0000 - val_tn0.9: 2482776.0000 - val_fn0.9: 217947.0000 - val_precis
    ion0.9: 0.2646 - val_recall0.9: 0.7289 - val_accuracy: 0.5776 - val_auc: 0.7124 - val_f1: 0.2811
    Epoch 3/20
    800/800 [==============================] - 23s 28ms/step - loss: 1.0116 - tp0.1: 2624849.0000 - fp0.1: 7645828.0000 - tn0.1: 9037140.0000 - fn0.1: 352983.0000 - precision0.1: 0.2556 - reca
    ll0.1: 0.8815 - tp0.3: 1498713.0000 - fp0.3: 1695358.0000 - tn0.3: 14987610.0000 - fn0.3: 1479119.0000 - precision0.3: 0.4692 - recall0.3: 0.5033 - tp0.5: 975584.0000 - fp0.5: 420254.0000
    - tn0.5: 16262714.0000 - fn0.5: 2002248.0000 - precision0.5: 0.6989 - recall0.5: 0.3276 - tp0.7: 709358.0000 - fp0.7: 149271.0000 - tn0.7: 16533697.0000 - fn0.7: 2268474.0000 - precision0.
    7: 0.8262 - recall0.7: 0.2382 - tp0.9: 389012.0000 - fp0.9: 27027.0000 - tn0.9: 16655941.0000 - fn0.9: 2588820.0000 - precision0.9: 0.9350 - recall0.9: 0.1306 - accuracy: 0.8768 - auc: 0.8
    104 - f1: 0.2631 - val_loss: 1.5866 - val_tp0.1: 541567.0000 - val_fp0.1: 1371385.0000 - val_tn0.1: 2739958.0000 - val_fn0.1: 262290.0000 - val_precision0.1: 0.2831 - val_recall0.1: 0.6737
     - val_tp0.3: 350430.0000 - val_fp0.3: 695890.0000 - val_tn0.3: 3415453.0000 - val_fn0.3: 453427.0000 - val_precision0.3: 0.3349 - val_recall0.3: 0.4359 - val_tp0.5: 277275.0000 - val_fp0.
    5: 441168.0000 - val_tn0.5: 3670175.0000 - val_fn0.5: 526582.0000 - val_precision0.5: 0.3859 - val_recall0.5: 0.3449 - val_tp0.7: 207293.0000 - val_fp0.7: 222426.0000 - val_tn0.7: 3888917.
    0000 - val_fn0.7: 596564.0000 - val_precision0.7: 0.4824 - val_recall0.7: 0.2579 - val_tp0.9: 90589.0000 - val_fp0.9: 35047.0000 - val_tn0.9: 4076296.0000 - val_fn0.9: 713268.0000 - val_pr
    ecision0.9: 0.7210 - val_recall0.9: 0.1127 - val_accuracy: 0.8031 - val_auc: 0.7058 - val_f1: 0.2811
    Epoch 4/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.9534 - tp0.1: 2630735.0000 - fp0.1: 7252035.0000 - tn0.1: 9430933.0000 - fn0.1: 347097.0000 - precision0.1: 0.2662 - reca
    ll0.1: 0.8834 - tp0.3: 1613584.0000 - fp0.3: 1430388.0000 - tn0.3: 15252580.0000 - fn0.3: 1364248.0000 - precision0.3: 0.5301 - recall0.3: 0.5419 - tp0.5: 1188917.0000 - fp0.5: 484302.0000
     - tn0.5: 16198666.0000 - fn0.5: 1788915.0000 - precision0.5: 0.7106 - recall0.5: 0.3993 - tp0.7: 896123.0000 - fp0.7: 181140.0000 - tn0.7: 16501828.0000 - fn0.7: 2081709.0000 - precision0
    .7: 0.8319 - recall0.7: 0.3009 - tp0.9: 494120.0000 - fp0.9: 27734.0000 - tn0.9: 16655234.0000 - fn0.9: 2483712.0000 - precision0.9: 0.9469 - recall0.9: 0.1659 - accuracy: 0.8844 - auc: 0.
    8330 - f1: 0.2631 - val_loss: 1.2238 - val_tp0.1: 762287.0000 - val_fp0.1: 2841918.0000 - val_tn0.1: 1269425.0000 - val_fn0.1: 41570.0000 - val_precision0.1: 0.2115 - val_recall0.1: 0.9483
     - val_tp0.3: 473341.0000 - val_fp0.3: 934680.0000 - val_tn0.3: 3176663.0000 - val_fn0.3: 330516.0000 - val_precision0.3: 0.3362 - val_recall0.3: 0.5888 - val_tp0.5: 284941.0000 - val_fp0.
    5: 382747.0000 - val_tn0.5: 3728596.0000 - val_fn0.5: 518916.0000 - val_precision0.5: 0.4268 - val_recall0.5: 0.3545 - val_tp0.7: 199426.0000 - val_fp0.7: 215679.0000 - val_tn0.7: 3895664.
    0000 - val_fn0.7: 604431.0000 - val_precision0.7: 0.4804 - val_recall0.7: 0.2481 - val_tp0.9: 80714.0000 - val_fp0.9: 40538.0000 - val_tn0.9: 4070805.0000 - val_fn0.9: 723143.0000 - val_pr
    ecision0.9: 0.6657 - val_recall0.9: 0.1004 - val_accuracy: 0.8166 - val_auc: 0.7536 - val_f1: 0.2811
    Epoch 5/20
    800/800 [==============================] - 105s 132ms/step - loss: 0.9031 - tp0.1: 2615817.0000 - fp0.1: 6473866.0000 - tn0.1: 10209102.0000 - fn0.1: 362015.0000 - precision0.1: 0.2878 - r
    ecall0.1: 0.8784 - tp0.3: 1760338.0000 - fp0.3: 1552636.0000 - tn0.3: 15130332.0000 - fn0.3: 1217494.0000 - precision0.3: 0.5313 - recall0.3: 0.5911 - tp0.5: 1305140.0000 - fp0.5: 538907.0
    000 - tn0.5: 16144061.0000 - fn0.5: 1672692.0000 - precision0.5: 0.7078 - recall0.5: 0.4383 - tp0.7: 972649.0000 - fp0.7: 195218.0000 - tn0.7: 16487750.0000 - fn0.7: 2005183.0000 - precisi
    on0.7: 0.8328 - recall0.7: 0.3266 - tp0.9: 541205.0000 - fp0.9: 31289.0000 - tn0.9: 16651679.0000 - fn0.9: 2436627.0000 - precision0.9: 0.9453 - recall0.9: 0.1817 - accuracy: 0.8875 - auc:
     0.8476 - f1: 0.2631 - val_loss: 1.0983 - val_tp0.1: 685124.0000 - val_fp0.1: 1862694.0000 - val_tn0.1: 2248649.0000 - val_fn0.1: 118733.0000 - val_precision0.1: 0.2689 - val_recall0.1: 0.
    8523 - val_tp0.3: 408907.0000 - val_fp0.3: 510069.0000 - val_tn0.3: 3601274.0000 - val_fn0.3: 394950.0000 - val_precision0.3: 0.4450 - val_recall0.3: 0.5087 - val_tp0.5: 249171.0000 - val_
    fp0.5: 222225.0000 - val_tn0.5: 3889118.0000 - val_fn0.5: 554686.0000 - val_precision0.5: 0.5286 - val_recall0.5: 0.3100 - val_tp0.7: 157112.0000 - val_fp0.7: 77659.0000 - val_tn0.7: 40336
    84.0000 - val_fn0.7: 646745.0000 - val_precision0.7: 0.6692 - val_recall0.7: 0.1954 - val_tp0.9: 31306.0000 - val_fp0.9: 406.0000 - val_tn0.9: 4110937.0000 - val_fn0.9: 772551.0000 - val_p
    recision0.9: 0.9872 - val_recall0.9: 0.0389 - val_accuracy: 0.8419 - val_auc: 0.7829 - val_f1: 0.2811
    Epoch 6/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.8664 - tp0.1: 2613105.0000 - fp0.1: 5967389.0000 - tn0.1: 10715579.0000 - fn0.1: 364727.0000 - precision0.1: 0.3045 - rec
    all0.1: 0.8775 - tp0.3: 1833388.0000 - fp0.3: 1537464.0000 - tn0.3: 15145504.0000 - fn0.3: 1144444.0000 - precision0.3: 0.5439 - recall0.3: 0.6157 - tp0.5: 1388900.0000 - fp0.5: 539537.000
    0 - tn0.5: 16143431.0000 - fn0.5: 1588932.0000 - precision0.5: 0.7202 - recall0.5: 0.4664 - tp0.7: 1047260.0000 - fp0.7: 195978.0000 - tn0.7: 16486990.0000 - fn0.7: 1930572.0000 - precisio
    n0.7: 0.8424 - recall0.7: 0.3517 - tp0.9: 612618.0000 - fp0.9: 34165.0000 - tn0.9: 16648803.0000 - fn0.9: 2365214.0000 - precision0.9: 0.9472 - recall0.9: 0.2057 - accuracy: 0.8917 - auc:
    0.8558 - f1: 0.2631 - val_loss: 1.1906 - val_tp0.1: 589351.0000 - val_fp0.1: 1203883.0000 - val_tn0.1: 2907460.0000 - val_fn0.1: 214506.0000 - val_precision0.1: 0.3287 - val_recall0.1: 0.7
    332 - val_tp0.3: 396327.0000 - val_fp0.3: 557062.0000 - val_tn0.3: 3554281.0000 - val_fn0.3: 407530.0000 - val_precision0.3: 0.4157 - val_recall0.3: 0.4930 - val_tp0.5: 335193.0000 - val_f
    p0.5: 411860.0000 - val_tn0.5: 3699483.0000 - val_fn0.5: 468664.0000 - val_precision0.5: 0.4487 - val_recall0.5: 0.4170 - val_tp0.7: 273790.0000 - val_fp0.7: 273418.0000 - val_tn0.7: 38379
    25.0000 - val_fn0.7: 530067.0000 - val_precision0.7: 0.5003 - val_recall0.7: 0.3406 - val_tp0.9: 147281.0000 - val_fp0.9: 47498.0000 - val_tn0.9: 4063845.0000 - val_fn0.9: 656576.0000 - va
    l_precision0.9: 0.7561 - val_recall0.9: 0.1832 - val_accuracy: 0.8209 - val_auc: 0.7717 - val_f1: 0.2811
    Epoch 7/20
    800/800 [==============================] - 23s 29ms/step - loss: 0.8261 - tp0.1: 2610067.0000 - fp0.1: 5441450.0000 - tn0.1: 11241518.0000 - fn0.1: 367765.0000 - precision0.1: 0.3242 - rec
    all0.1: 0.8765 - tp0.3: 1926083.0000 - fp0.3: 1638761.0000 - tn0.3: 15044207.0000 - fn0.3: 1051749.0000 - precision0.3: 0.5403 - recall0.3: 0.6468 - tp0.5: 1451422.0000 - fp0.5: 566878.000
    0 - tn0.5: 16116090.0000 - fn0.5: 1526410.0000 - precision0.5: 0.7191 - recall0.5: 0.4874 - tp0.7: 1084565.0000 - fp0.7: 195437.0000 - tn0.7: 16487531.0000 - fn0.7: 1893267.0000 - precisio
    n0.7: 0.8473 - recall0.7: 0.3642 - tp0.9: 629952.0000 - fp0.9: 32457.0000 - tn0.9: 16650511.0000 - fn0.9: 2347880.0000 - precision0.9: 0.9510 - recall0.9: 0.2115 - accuracy: 0.8935 - auc:
    0.8646 - f1: 0.2631 - val_loss: 0.9781 - val_tp0.1: 706790.0000 - val_fp0.1: 1645956.0000 - val_tn0.1: 2465387.0000 - val_fn0.1: 97067.0000 - val_precision0.1: 0.3004 - val_recall0.1: 0.87
    92 - val_tp0.3: 517245.0000 - val_fp0.3: 647485.0000 - val_tn0.3: 3463858.0000 - val_fn0.3: 286612.0000 - val_precision0.3: 0.4441 - val_recall0.3: 0.6435 - val_tp0.5: 373721.0000 - val_fp
    0.5: 255513.0000 - val_tn0.5: 3855830.0000 - val_fn0.5: 430136.0000 - val_precision0.5: 0.5939 - val_recall0.5: 0.4649 - val_tp0.7: 242514.0000 - val_fp0.7: 84275.0000 - val_tn0.7: 4027068
    .0000 - val_fn0.7: 561343.0000 - val_precision0.7: 0.7421 - val_recall0.7: 0.3017 - val_tp0.9: 127788.0000 - val_fp0.9: 15373.0000 - val_tn0.9: 4095970.0000 - val_fn0.9: 676069.0000 - val_
    precision0.9: 0.8926 - val_recall0.9: 0.1590 - val_accuracy: 0.8605 - val_auc: 0.8289 - val_f1: 0.2811
    Epoch 8/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.7968 - tp0.1: 2628693.0000 - fp0.1: 5237949.0000 - tn0.1: 11445019.0000 - fn0.1: 349139.0000 - precision0.1: 0.3342 - rec
    all0.1: 0.8828 - tp0.3: 1990774.0000 - fp0.3: 1601561.0000 - tn0.3: 15081407.0000 - fn0.3: 987058.0000 - precision0.3: 0.5542 - recall0.3: 0.6685 - tp0.5: 1512649.0000 - fp0.5: 585437.0000
     - tn0.5: 16097531.0000 - fn0.5: 1465183.0000 - precision0.5: 0.7210 - recall0.5: 0.5080 - tp0.7: 1113359.0000 - fp0.7: 192219.0000 - tn0.7: 16490749.0000 - fn0.7: 1864473.0000 - precision
    0.7: 0.8528 - recall0.7: 0.3739 - tp0.9: 662809.0000 - fp0.9: 30877.0000 - tn0.9: 16652091.0000 - fn0.9: 2315023.0000 - precision0.9: 0.9555 - recall0.9: 0.2226 - accuracy: 0.8957 - auc: 0
    .8732 - f1: 0.2631 - val_loss: 1.0262 - val_tp0.1: 703581.0000 - val_fp0.1: 1905662.0000 - val_tn0.1: 2205681.0000 - val_fn0.1: 100276.0000 - val_precision0.1: 0.2696 - val_recall0.1: 0.87
    53 - val_tp0.3: 447413.0000 - val_fp0.3: 447251.0000 - val_tn0.3: 3664092.0000 - val_fn0.3: 356444.0000 - val_precision0.3: 0.5001 - val_recall0.3: 0.5566 - val_tp0.5: 293285.0000 - val_fp
    0.5: 134465.0000 - val_tn0.5: 3976878.0000 - val_fn0.5: 510572.0000 - val_precision0.5: 0.6856 - val_recall0.5: 0.3648 - val_tp0.7: 180739.0000 - val_fp0.7: 40494.0000 - val_tn0.7: 4070849
    .0000 - val_fn0.7: 623118.0000 - val_precision0.7: 0.8170 - val_recall0.7: 0.2248 - val_tp0.9: 86388.0000 - val_fp0.9: 5793.0000 - val_tn0.9: 4105550.0000 - val_fn0.9: 717469.0000 - val_pr
    ecision0.9: 0.9372 - val_recall0.9: 0.1075 - val_accuracy: 0.8688 - val_auc: 0.8159 - val_f1: 0.2811
    Epoch 9/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.7819 - tp0.1: 2640898.0000 - fp0.1: 5239761.0000 - tn0.1: 11443207.0000 - fn0.1: 336934.0000 - precision0.1: 0.3351 - rec
    all0.1: 0.8869 - tp0.3: 2019563.0000 - fp0.3: 1566552.0000 - tn0.3: 15116416.0000 - fn0.3: 958269.0000 - precision0.3: 0.5632 - recall0.3: 0.6782 - tp0.5: 1571033.0000 - fp0.5: 592615.0000
     - tn0.5: 16090353.0000 - fn0.5: 1406799.0000 - precision0.5: 0.7261 - recall0.5: 0.5276 - tp0.7: 1175144.0000 - fp0.7: 194680.0000 - tn0.7: 16488288.0000 - fn0.7: 1802688.0000 - precision
    0.7: 0.8579 - recall0.7: 0.3946 - tp0.9: 742669.0000 - fp0.9: 35986.0000 - tn0.9: 16646982.0000 - fn0.9: 2235163.0000 - precision0.9: 0.9538 - recall0.9: 0.2494 - accuracy: 0.8983 - auc: 0
    .8780 - f1: 0.2631 - val_loss: 1.0157 - val_tp0.1: 652637.0000 - val_fp0.1: 1329883.0000 - val_tn0.1: 2781460.0000 - val_fn0.1: 151220.0000 - val_precision0.1: 0.3292 - val_recall0.1: 0.81
    19 - val_tp0.3: 420948.0000 - val_fp0.3: 353053.0000 - val_tn0.3: 3758290.0000 - val_fn0.3: 382909.0000 - val_precision0.3: 0.5439 - val_recall0.3: 0.5237 - val_tp0.5: 281081.0000 - val_fp
    0.5: 123121.0000 - val_tn0.5: 3988222.0000 - val_fn0.5: 522776.0000 - val_precision0.5: 0.6954 - val_recall0.5: 0.3497 - val_tp0.7: 173940.0000 - val_fp0.7: 36200.0000 - val_tn0.7: 4075143
    .0000 - val_fn0.7: 629917.0000 - val_precision0.7: 0.8277 - val_recall0.7: 0.2164 - val_tp0.9: 89015.0000 - val_fp0.9: 6773.0000 - val_tn0.9: 4104570.0000 - val_fn0.9: 714842.0000 - val_pr
    ecision0.9: 0.9293 - val_recall0.9: 0.1107 - val_accuracy: 0.8686 - val_auc: 0.8166 - val_f1: 0.2811
    Epoch 10/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.7672 - tp0.1: 2635701.0000 - fp0.1: 4919057.0000 - tn0.1: 11763911.0000 - fn0.1: 342131.0000 - precision0.1: 0.3489 - rec
    all0.1: 0.8851 - tp0.3: 2036938.0000 - fp0.3: 1575677.0000 - tn0.3: 15107291.0000 - fn0.3: 940894.0000 - precision0.3: 0.5638 - recall0.3: 0.6840 - tp0.5: 1588676.0000 - fp0.5: 624407.0000
     - tn0.5: 16058561.0000 - fn0.5: 1389156.0000 - precision0.5: 0.7179 - recall0.5: 0.5335 - tp0.7: 1150076.0000 - fp0.7: 189388.0000 - tn0.7: 16493580.0000 - fn0.7: 1827756.0000 - precision
    0.7: 0.8586 - recall0.7: 0.3862 - tp0.9: 694637.0000 - fp0.9: 31558.0000 - tn0.9: 16651410.0000 - fn0.9: 2283195.0000 - precision0.9: 0.9565 - recall0.9: 0.2333 - accuracy: 0.8976 - auc: 0
    .8799 - f1: 0.2631 - val_loss: 0.9685 - val_tp0.1: 719258.0000 - val_fp0.1: 1852284.0000 - val_tn0.1: 2259059.0000 - val_fn0.1: 84599.0000 - val_precision0.1: 0.2797 - val_recall0.1: 0.894
    8 - val_tp0.3: 495938.0000 - val_fp0.3: 497006.0000 - val_tn0.3: 3614337.0000 - val_fn0.3: 307919.0000 - val_precision0.3: 0.4995 - val_recall0.3: 0.6169 - val_tp0.5: 339199.0000 - val_fp0
    .5: 177289.0000 - val_tn0.5: 3934054.0000 - val_fn0.5: 464658.0000 - val_precision0.5: 0.6567 - val_recall0.5: 0.4220 - val_tp0.7: 196231.0000 - val_fp0.7: 44292.0000 - val_tn0.7: 4067051.
    0000 - val_fn0.7: 607626.0000 - val_precision0.7: 0.8159 - val_recall0.7: 0.2441 - val_tp0.9: 84223.0000 - val_fp0.9: 4726.0000 - val_tn0.9: 4106617.0000 - val_fn0.9: 719634.0000 - val_pre
    cision0.9: 0.9469 - val_recall0.9: 0.1048 - val_accuracy: 0.8694 - val_auc: 0.8365 - val_f1: 0.2811
    Epoch 11/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.7657 - tp0.1: 2620304.0000 - fp0.1: 4756693.0000 - tn0.1: 11926275.0000 - fn0.1: 357528.0000 - precision0.1: 0.3552 - rec
    all0.1: 0.8799 - tp0.3: 2057463.0000 - fp0.3: 1595402.0000 - tn0.3: 15087566.0000 - fn0.3: 920369.0000 - precision0.3: 0.5632 - recall0.3: 0.6909 - tp0.5: 1629457.0000 - fp0.5: 652845.0000
     - tn0.5: 16030123.0000 - fn0.5: 1348375.0000 - precision0.5: 0.7140 - recall0.5: 0.5472 - tp0.7: 1159302.0000 - fp0.7: 181594.0000 - tn0.7: 16501374.0000 - fn0.7: 1818530.0000 - precision
    0.7: 0.8646 - recall0.7: 0.3893 - tp0.9: 691411.0000 - fp0.9: 29139.0000 - tn0.9: 16653829.0000 - fn0.9: 2286421.0000 - precision0.9: 0.9596 - recall0.9: 0.2322 - accuracy: 0.8982 - auc: 0
    .8794 - f1: 0.2631 - val_loss: 0.9798 - val_tp0.1: 650390.0000 - val_fp0.1: 1145138.0000 - val_tn0.1: 2966205.0000 - val_fn0.1: 153467.0000 - val_precision0.1: 0.3622 - val_recall0.1: 0.80
    91 - val_tp0.3: 416815.0000 - val_fp0.3: 345286.0000 - val_tn0.3: 3766057.0000 - val_fn0.3: 387042.0000 - val_precision0.3: 0.5469 - val_recall0.3: 0.5185 - val_tp0.5: 303813.0000 - val_fp
    0.5: 119881.0000 - val_tn0.5: 3991462.0000 - val_fn0.5: 500044.0000 - val_precision0.5: 0.7171 - val_recall0.5: 0.3779 - val_tp0.7: 186405.0000 - val_fp0.7: 34348.0000 - val_tn0.7: 4076995
    .0000 - val_fn0.7: 617452.0000 - val_precision0.7: 0.8444 - val_recall0.7: 0.2319 - val_tp0.9: 88035.0000 - val_fp0.9: 4840.0000 - val_tn0.9: 4106503.0000 - val_fn0.9: 715822.0000 - val_pr
    ecision0.9: 0.9479 - val_recall0.9: 0.1095 - val_accuracy: 0.8739 - val_auc: 0.8236 - val_f1: 0.2811
    Epoch 12/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.7483 - tp0.1: 2643606.0000 - fp0.1: 4772914.0000 - tn0.1: 11910054.0000 - fn0.1: 334226.0000 - precision0.1: 0.3564 - rec
    all0.1: 0.8878 - tp0.3: 2076874.0000 - fp0.3: 1529573.0000 - tn0.3: 15153395.0000 - fn0.3: 900958.0000 - precision0.3: 0.5759 - recall0.3: 0.6974 - tp0.5: 1673377.0000 - fp0.5: 652544.0000
     - tn0.5: 16030424.0000 - fn0.5: 1304455.0000 - precision0.5: 0.7194 - recall0.5: 0.5619 - tp0.7: 1197669.0000 - fp0.7: 195301.0000 - tn0.7: 16487667.0000 - fn0.7: 1780163.0000 - precision
    0.7: 0.8598 - recall0.7: 0.4022 - tp0.9: 723663.0000 - fp0.9: 34095.0000 - tn0.9: 16648873.0000 - fn0.9: 2254169.0000 - precision0.9: 0.9550 - recall0.9: 0.2430 - accuracy: 0.9005 - auc: 0
    .8854 - f1: 0.2631 - val_loss: 0.9661 - val_tp0.1: 682417.0000 - val_fp0.1: 1411604.0000 - val_tn0.1: 2699739.0000 - val_fn0.1: 121440.0000 - val_precision0.1: 0.3259 - val_recall0.1: 0.84
    89 - val_tp0.3: 393462.0000 - val_fp0.3: 248966.0000 - val_tn0.3: 3862377.0000 - val_fn0.3: 410395.0000 - val_precision0.3: 0.6125 - val_recall0.3: 0.4895 - val_tp0.5: 264605.0000 - val_fp
    0.5: 75544.0000 - val_tn0.5: 4035799.0000 - val_fn0.5: 539252.0000 - val_precision0.5: 0.7779 - val_recall0.5: 0.3292 - val_tp0.7: 151048.0000 - val_fp0.7: 18110.0000 - val_tn0.7: 4093233.
    0000 - val_fn0.7: 652809.0000 - val_precision0.7: 0.8929 - val_recall0.7: 0.1879 - val_tp0.9: 73096.0000 - val_fp0.9: 2503.0000 - val_tn0.9: 4108840.0000 - val_fn0.9: 730761.0000 - val_pre
    cision0.9: 0.9669 - val_recall0.9: 0.0909 - val_accuracy: 0.8749 - val_auc: 0.8348 - val_f1: 0.2811
    Epoch 13/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.7469 - tp0.1: 2638319.0000 - fp0.1: 4684335.0000 - tn0.1: 11998633.0000 - fn0.1: 339513.0000 - precision0.1: 0.3603 - rec
    all0.1: 0.8860 - tp0.3: 2075631.0000 - fp0.3: 1533614.0000 - tn0.3: 15149354.0000 - fn0.3: 902201.0000 - precision0.3: 0.5751 - recall0.3: 0.6970 - tp0.5: 1654093.0000 - fp0.5: 657052.0000
     - tn0.5: 16025916.0000 - fn0.5: 1323739.0000 - precision0.5: 0.7157 - recall0.5: 0.5555 - tp0.7: 1159144.0000 - fp0.7: 187558.0000 - tn0.7: 16495410.0000 - fn0.7: 1818688.0000 - precision
    0.7: 0.8607 - recall0.7: 0.3893 - tp0.9: 691098.0000 - fp0.9: 30329.0000 - tn0.9: 16652639.0000 - fn0.9: 2286734.0000 - precision0.9: 0.9580 - recall0.9: 0.2321 - accuracy: 0.8993 - auc: 0
    .8851 - f1: 0.2631 - val_loss: 0.9511 - val_tp0.1: 676012.0000 - val_fp0.1: 1300131.0000 - val_tn0.1: 2811212.0000 - val_fn0.1: 127845.0000 - val_precision0.1: 0.3421 - val_recall0.1: 0.84
    10 - val_tp0.3: 435613.0000 - val_fp0.3: 369776.0000 - val_tn0.3: 3741567.0000 - val_fn0.3: 368244.0000 - val_precision0.3: 0.5409 - val_recall0.3: 0.5419 - val_tp0.5: 315382.0000 - val_fp
    0.5: 126324.0000 - val_tn0.5: 3985019.0000 - val_fn0.5: 488475.0000 - val_precision0.5: 0.7140 - val_recall0.5: 0.3923 - val_tp0.7: 186834.0000 - val_fp0.7: 33643.0000 - val_tn0.7: 4077700
    .0000 - val_fn0.7: 617023.0000 - val_precision0.7: 0.8474 - val_recall0.7: 0.2324 - val_tp0.9: 86226.0000 - val_fp0.9: 4347.0000 - val_tn0.9: 4106996.0000 - val_fn0.9: 717631.0000 - val_pr
    ecision0.9: 0.9520 - val_recall0.9: 0.1073 - val_accuracy: 0.8749 - val_auc: 0.8317 - val_f1: 0.2811
    Epoch 14/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.7472 - tp0.1: 2640336.0000 - fp0.1: 4682320.0000 - tn0.1: 12000648.0000 - fn0.1: 337496.0000 - precision0.1: 0.3606 - rec
    all0.1: 0.8867 - tp0.3: 2072009.0000 - fp0.3: 1595973.0000 - tn0.3: 15086995.0000 - fn0.3: 905823.0000 - precision0.3: 0.5649 - recall0.3: 0.6958 - tp0.5: 1664556.0000 - fp0.5: 686406.0000
     - tn0.5: 15996562.0000 - fn0.5: 1313276.0000 - precision0.5: 0.7080 - recall0.5: 0.5590 - tp0.7: 1168681.0000 - fp0.7: 190677.0000 - tn0.7: 16492291.0000 - fn0.7: 1809151.0000 - precision
    0.7: 0.8597 - recall0.7: 0.3925 - tp0.9: 706963.0000 - fp0.9: 30254.0000 - tn0.9: 16652714.0000 - fn0.9: 2270869.0000 - precision0.9: 0.9590 - recall0.9: 0.2374 - accuracy: 0.8983 - auc: 0
    .8838 - f1: 0.2631 - val_loss: 0.9537 - val_tp0.1: 687713.0000 - val_fp0.1: 1442587.0000 - val_tn0.1: 2668756.0000 - val_fn0.1: 116144.0000 - val_precision0.1: 0.3228 - val_recall0.1: 0.85
    55 - val_tp0.3: 448956.0000 - val_fp0.3: 376999.0000 - val_tn0.3: 3734344.0000 - val_fn0.3: 354901.0000 - val_precision0.3: 0.5436 - val_recall0.3: 0.5585 - val_tp0.5: 322122.0000 - val_fp
    0.5: 133228.0000 - val_tn0.5: 3978115.0000 - val_fn0.5: 481735.0000 - val_precision0.5: 0.7074 - val_recall0.5: 0.4007 - val_tp0.7: 189079.0000 - val_fp0.7: 36378.0000 - val_tn0.7: 4074965
    .0000 - val_fn0.7: 614778.0000 - val_precision0.7: 0.8386 - val_recall0.7: 0.2352 - val_tp0.9: 87530.0000 - val_fp0.9: 4845.0000 - val_tn0.9: 4106498.0000 - val_fn0.9: 716327.0000 - val_pr
    ecision0.9: 0.9476 - val_recall0.9: 0.1089 - val_accuracy: 0.8749 - val_auc: 0.8339 - val_f1: 0.2811
    Epoch 15/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.7436 - tp0.1: 2645410.0000 - fp0.1: 4733153.0000 - tn0.1: 11949815.0000 - fn0.1: 332422.0000 - precision0.1: 0.3585 - rec
    all0.1: 0.8884 - tp0.3: 2099091.0000 - fp0.3: 1622697.0000 - tn0.3: 15060271.0000 - fn0.3: 878741.0000 - precision0.3: 0.5640 - recall0.3: 0.7049 - tp0.5: 1689821.0000 - fp0.5: 706002.0000
     - tn0.5: 15976966.0000 - fn0.5: 1288011.0000 - precision0.5: 0.7053 - recall0.5: 0.5675 - tp0.7: 1187271.0000 - fp0.7: 194662.0000 - tn0.7: 16488306.0000 - fn0.7: 1790561.0000 - precision
    0.7: 0.8591 - recall0.7: 0.3987 - tp0.9: 719432.0000 - fp0.9: 31350.0000 - tn0.9: 16651618.0000 - fn0.9: 2258400.0000 - precision0.9: 0.9582 - recall0.9: 0.2416 - accuracy: 0.8986 - auc: 0
    .8855 - f1: 0.2631 - val_loss: 0.9517 - val_tp0.1: 686003.0000 - val_fp0.1: 1413980.0000 - val_tn0.1: 2697363.0000 - val_fn0.1: 117854.0000 - val_precision0.1: 0.3267 - val_recall0.1: 0.85
    34 - val_tp0.3: 449760.0000 - val_fp0.3: 382567.0000 - val_tn0.3: 3728776.0000 - val_fn0.3: 354097.0000 - val_precision0.3: 0.5404 - val_recall0.3: 0.5595 - val_tp0.5: 324012.0000 - val_fp
    0.5: 130173.0000 - val_tn0.5: 3981170.0000 - val_fn0.5: 479845.0000 - val_precision0.5: 0.7134 - val_recall0.5: 0.4031 - val_tp0.7: 189632.0000 - val_fp0.7: 34603.0000 - val_tn0.7: 4076740
    .0000 - val_fn0.7: 614225.0000 - val_precision0.7: 0.8457 - val_recall0.7: 0.2359 - val_tp0.9: 86754.0000 - val_fp0.9: 4383.0000 - val_tn0.9: 4106960.0000 - val_fn0.9: 717103.0000 - val_pr
    ecision0.9: 0.9519 - val_recall0.9: 0.1079 - val_accuracy: 0.8759 - val_auc: 0.8345 - val_f1: 0.2811
    Epoch 16/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.7458 - tp0.1: 2641645.0000 - fp0.1: 4695396.0000 - tn0.1: 11987572.0000 - fn0.1: 336187.0000 - precision0.1: 0.3600 - rec
    all0.1: 0.8871 - tp0.3: 2084584.0000 - fp0.3: 1588554.0000 - tn0.3: 15094414.0000 - fn0.3: 893248.0000 - precision0.3: 0.5675 - recall0.3: 0.7000 - tp0.5: 1683391.0000 - fp0.5: 686943.0000
     - tn0.5: 15996025.0000 - fn0.5: 1294441.0000 - precision0.5: 0.7102 - recall0.5: 0.5653 - tp0.7: 1187166.0000 - fp0.7: 191714.0000 - tn0.7: 16491254.0000 - fn0.7: 1790666.0000 - precision
    0.7: 0.8610 - recall0.7: 0.3987 - tp0.9: 713157.0000 - fp0.9: 30164.0000 - tn0.9: 16652804.0000 - fn0.9: 2264675.0000 - precision0.9: 0.9594 - recall0.9: 0.2395 - accuracy: 0.8992 - auc: 0
    .8851 - f1: 0.2631 - val_loss: 0.9480 - val_tp0.1: 687202.0000 - val_fp0.1: 1416765.0000 - val_tn0.1: 2694578.0000 - val_fn0.1: 116655.0000 - val_precision0.1: 0.3266 - val_recall0.1: 0.85
    49 - val_tp0.3: 435185.0000 - val_fp0.3: 324375.0000 - val_tn0.3: 3786968.0000 - val_fn0.3: 368672.0000 - val_precision0.3: 0.5729 - val_recall0.3: 0.5414 - val_tp0.5: 310697.0000 - val_fp
    0.5: 114008.0000 - val_tn0.5: 3997335.0000 - val_fn0.5: 493160.0000 - val_precision0.5: 0.7316 - val_recall0.5: 0.3865 - val_tp0.7: 180896.0000 - val_fp0.7: 30514.0000 - val_tn0.7: 4080829
    .0000 - val_fn0.7: 622961.0000 - val_precision0.7: 0.8557 - val_recall0.7: 0.2250 - val_tp0.9: 85121.0000 - val_fp0.9: 4248.0000 - val_tn0.9: 4107095.0000 - val_fn0.9: 718736.0000 - val_pr
    ecision0.9: 0.9525 - val_recall0.9: 0.1059 - val_accuracy: 0.8765 - val_auc: 0.8372 - val_f1: 0.2811
    Epoch 17/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.7489 - tp0.1: 2637863.0000 - fp0.1: 4686124.0000 - tn0.1: 11996844.0000 - fn0.1: 339969.0000 - precision0.1: 0.3602 - rec
    all0.1: 0.8858 - tp0.3: 2075559.0000 - fp0.3: 1564588.0000 - tn0.3: 15118380.0000 - fn0.3: 902273.0000 - precision0.3: 0.5702 - recall0.3: 0.6970 - tp0.5: 1674724.0000 - fp0.5: 689631.0000
     - tn0.5: 15993337.0000 - fn0.5: 1303108.0000 - precision0.5: 0.7083 - recall0.5: 0.5624 - tp0.7: 1179885.0000 - fp0.7: 191863.0000 - tn0.7: 16491105.0000 - fn0.7: 1797947.0000 - precision
    0.7: 0.8601 - recall0.7: 0.3962 - tp0.9: 710902.0000 - fp0.9: 31154.0000 - tn0.9: 16651814.0000 - fn0.9: 2266930.0000 - precision0.9: 0.9580 - recall0.9: 0.2387 - accuracy: 0.8986 - auc: 0
    .8842 - f1: 0.2631 - val_loss: 0.9526 - val_tp0.1: 685383.0000 - val_fp0.1: 1407315.0000 - val_tn0.1: 2704028.0000 - val_fn0.1: 118474.0000 - val_precision0.1: 0.3275 - val_recall0.1: 0.85
    26 - val_tp0.3: 434253.0000 - val_fp0.3: 334816.0000 - val_tn0.3: 3776527.0000 - val_fn0.3: 369604.0000 - val_precision0.3: 0.5646 - val_recall0.3: 0.5402 - val_tp0.5: 309436.0000 - val_fp
    0.5: 113860.0000 - val_tn0.5: 3997483.0000 - val_fn0.5: 494421.0000 - val_precision0.5: 0.7310 - val_recall0.5: 0.3849 - val_tp0.7: 180756.0000 - val_fp0.7: 30132.0000 - val_tn0.7: 4081211
    .0000 - val_fn0.7: 623101.0000 - val_precision0.7: 0.8571 - val_recall0.7: 0.2249 - val_tp0.9: 85627.0000 - val_fp0.9: 4210.0000 - val_tn0.9: 4107133.0000 - val_fn0.9: 718230.0000 - val_pr
    ecision0.9: 0.9531 - val_recall0.9: 0.1065 - val_accuracy: 0.8762 - val_auc: 0.8351 - val_f1: 0.2811
    Epoch 18/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.7393 - tp0.1: 2648936.0000 - fp0.1: 4669640.0000 - tn0.1: 12013328.0000 - fn0.1: 328896.0000 - precision0.1: 0.3619 - rec
    all0.1: 0.8896 - tp0.3: 2088022.0000 - fp0.3: 1535327.0000 - tn0.3: 15147641.0000 - fn0.3: 889810.0000 - precision0.3: 0.5763 - recall0.3: 0.7012 - tp0.5: 1675010.0000 - fp0.5: 662872.0000
     - tn0.5: 16020096.0000 - fn0.5: 1302822.0000 - precision0.5: 0.7165 - recall0.5: 0.5625 - tp0.7: 1182736.0000 - fp0.7: 184372.0000 - tn0.7: 16498596.0000 - fn0.7: 1795096.0000 - precision
    0.7: 0.8651 - recall0.7: 0.3972 - tp0.9: 719695.0000 - fp0.9: 31350.0000 - tn0.9: 16651618.0000 - fn0.9: 2258137.0000 - precision0.9: 0.9583 - recall0.9: 0.2417 - accuracy: 0.9000 - auc: 0
    .8873 - f1: 0.2631 - val_loss: 0.9553 - val_tp0.1: 682153.0000 - val_fp0.1: 1382728.0000 - val_tn0.1: 2728615.0000 - val_fn0.1: 121704.0000 - val_precision0.1: 0.3304 - val_recall0.1: 0.84
    86 - val_tp0.3: 433097.0000 - val_fp0.3: 334545.0000 - val_tn0.3: 3776798.0000 - val_fn0.3: 370760.0000 - val_precision0.3: 0.5642 - val_recall0.3: 0.5388 - val_tp0.5: 307911.0000 - val_fp
    0.5: 113008.0000 - val_tn0.5: 3998335.0000 - val_fn0.5: 495946.0000 - val_precision0.5: 0.7315 - val_recall0.5: 0.3830 - val_tp0.7: 178664.0000 - val_fp0.7: 29733.0000 - val_tn0.7: 4081610
    .0000 - val_fn0.7: 625193.0000 - val_precision0.7: 0.8573 - val_recall0.7: 0.2223 - val_tp0.9: 83772.0000 - val_fp0.9: 4111.0000 - val_tn0.9: 4107232.0000 - val_fn0.9: 720085.0000 - val_pr
    ecision0.9: 0.9532 - val_recall0.9: 0.1042 - val_accuracy: 0.8761 - val_auc: 0.8340 - val_f1: 0.2811
    Epoch 19/20
    800/800 [==============================] - 23s 28ms/step - loss: 0.7452 - tp0.1: 2646665.0000 - fp0.1: 4698587.0000 - tn0.1: 11984381.0000 - fn0.1: 331167.0000 - precision0.1: 0.3603 - rec
    all0.1: 0.8888 - tp0.3: 2062641.0000 - fp0.3: 1557707.0000 - tn0.3: 15125261.0000 - fn0.3: 915191.0000 - precision0.3: 0.5697 - recall0.3: 0.6927 - tp0.5: 1648838.0000 - fp0.5: 680800.0000
     - tn0.5: 16002168.0000 - fn0.5: 1328994.0000 - precision0.5: 0.7078 - recall0.5: 0.5537 - tp0.7: 1164585.0000 - fp0.7: 186942.0000 - tn0.7: 16496026.0000 - fn0.7: 1813247.0000 - precision
    0.7: 0.8617 - recall0.7: 0.3911 - tp0.9: 705028.0000 - fp0.9: 30241.0000 - tn0.9: 16652727.0000 - fn0.9: 2272804.0000 - precision0.9: 0.9589 - recall0.9: 0.2368 - accuracy: 0.8978 - auc: 0
    .8850 - f1: 0.2631 - val_loss: 0.9544 - val_tp0.1: 689367.0000 - val_fp0.1: 1469991.0000 - val_tn0.1: 2641352.0000 - val_fn0.1: 114490.0000 - val_precision0.1: 0.3192 - val_recall0.1: 0.85
    76 - val_tp0.3: 451534.0000 - val_fp0.3: 363328.0000 - val_tn0.3: 3748015.0000 - val_fn0.3: 352323.0000 - val_precision0.3: 0.5541 - val_recall0.3: 0.5617 - val_tp0.5: 324284.0000 - val_fp
    0.5: 128372.0000 - val_tn0.5: 3982971.0000 - val_fn0.5: 479573.0000 - val_precision0.5: 0.7164 - val_recall0.5: 0.4034 - val_tp0.7: 190210.0000 - val_fp0.7: 35390.0000 - val_tn0.7: 4075953
    .0000 - val_fn0.7: 613647.0000 - val_precision0.7: 0.8431 - val_recall0.7: 0.2366 - val_tp0.9: 88933.0000 - val_fp0.9: 4874.0000 - val_tn0.9: 4106469.0000 - val_fn0.9: 714924.0000 - val_pr
    ecision0.9: 0.9480 - val_recall0.9: 0.1106 - val_accuracy: 0.8763 - val_auc: 0.8353 - val_f1: 0.2811
    Epoch 20/20
    800/800 [==============================] - 22s 28ms/step - loss: 0.7364 - tp0.1: 2654787.0000 - fp0.1: 4675725.0000 - tn0.1: 12007243.0000 - fn0.1: 323045.0000 - precision0.1: 0.3622 - rec
    all0.1: 0.8915 - tp0.3: 2081760.0000 - fp0.3: 1548676.0000 - tn0.3: 15134292.0000 - fn0.3: 896072.0000 - precision0.3: 0.5734 - recall0.3: 0.6991 - tp0.5: 1666321.0000 - fp0.5: 674703.0000
     - tn0.5: 16008265.0000 - fn0.5: 1311511.0000 - precision0.5: 0.7118 - recall0.5: 0.5596 - tp0.7: 1164069.0000 - fp0.7: 189776.0000 - tn0.7: 16493192.0000 - fn0.7: 1813763.0000 - precision
    0.7: 0.8598 - recall0.7: 0.3909 - tp0.9: 699952.0000 - fp0.9: 31138.0000 - tn0.9: 16651830.0000 - fn0.9: 2277880.0000 - precision0.9: 0.9574 - recall0.9: 0.2351 - accuracy: 0.8990 - auc: 0
    .8875 - f1: 0.2631 - val_loss: 0.9490 - val_tp0.1: 690953.0000 - val_fp0.1: 1466947.0000 - val_tn0.1: 2644396.0000 - val_fn0.1: 112904.0000 - val_precision0.1: 0.3202 - val_recall0.1: 0.85
    95 - val_tp0.3: 446418.0000 - val_fp0.3: 345263.0000 - val_tn0.3: 3766080.0000 - val_fn0.3: 357439.0000 - val_precision0.3: 0.5639 - val_recall0.3: 0.5553 - val_tp0.5: 321059.0000 - val_fp
    0.5: 123555.0000 - val_tn0.5: 3987788.0000 - val_fn0.5: 482798.0000 - val_precision0.5: 0.7221 - val_recall0.5: 0.3994 - val_tp0.7: 189576.0000 - val_fp0.7: 35232.0000 - val_tn0.7: 4076111
    .0000 - val_fn0.7: 614281.0000 - val_precision0.7: 0.8433 - val_recall0.7: 0.2358 - val_tp0.9: 90962.0000 - val_fp0.9: 5426.0000 - val_tn0.9: 4105917.0000 - val_fn0.9: 712895.0000 - val_pr
    ecision0.9: 0.9437 - val_recall0.9: 0.1132 - val_accuracy: 0.8766 - val_auc: 0.8372 - val_f1: 0.2811
    --- Running training session 61/140
    {'hp_epochs': 20, 'hp_batch_size': 20, 'hp_scaler': 'standard', 'hp_n_levels': 3, 'hp_first_filters': 128, 'hp_pool_size': 4, 'hp_input_size': 16384, 'hp_lr_start': 0.04354970735327304, 'h
    p_lr_power': 1.0}
    --- repeat #: 1
    input - shape:   (None, 16384, 1)
    output - shape:  (None, 16384, 1)
    Epoch 1/20
    240/240 [==============================] - 145s 530ms/step - loss: 0.7930 - tp0.1: 9902012.0000 - fp0.1: 12359941.0000 - tn0.1: 54313476.0000 - fn0.1: 2067755.0000 - precision0.1: 0.4448 -
     recall0.1: 0.8273 - tp0.3: 8813463.0000 - fp0.3: 7147978.0000 - tn0.3: 59525464.0000 - fn0.3: 3156304.0000 - precision0.3: 0.5522 - recall0.3: 0.7363 - tp0.5: 6775614.0000 - fp0.5: 236477
    6.0000 - tn0.5: 64308628.0000 - fn0.5: 5194153.0000 - precision0.5: 0.7413 - recall0.5: 0.5661 - tp0.7: 5269669.0000 - fp0.7: 804460.0000 - tn0.7: 65868980.0000 - fn0.7: 6700098.0000 - pre
    cision0.7: 0.8676 - recall0.7: 0.4402 - tp0.9: 3642391.0000 - fp0.9: 181910.0000 - tn0.9: 66491528.0000 - fn0.9: 8327376.0000 - precision0.9: 0.9524 - recall0.9: 0.3043 - accuracy: 0.9039
    - auc: 0.8736 - f1: 0.2642 - val_loss: 1.3831 - val_tp0.1: 1860092.0000 - val_fp0.1: 529606.0000 - val_tn0.1: 15968340.0000 - val_fn0.1: 1302762.0000 - val_precision0.1: 0.7784 - val_recal
    l0.1: 0.5881 - val_tp0.3: 1805849.0000 - val_fp0.3: 437765.0000 - val_tn0.3: 16060181.0000 - val_fn0.3: 1357005.0000 - val_precision0.3: 0.8049 - val_recall0.3: 0.5710 - val_tp0.5: 1737224
    .0000 - val_fp0.5: 360788.0000 - val_tn0.5: 16137158.0000 - val_fn0.5: 1425630.0000 - val_precision0.5: 0.8280 - val_recall0.5: 0.5493 - val_tp0.7: 1675344.0000 - val_fp0.7: 306812.0000 -
    val_tn0.7: 16191134.0000 - val_fn0.7: 1487510.0000 - val_precision0.7: 0.8452 - val_recall0.7: 0.5297 - val_tp0.9: 1555952.0000 - val_fp0.9: 227670.0000 - val_tn0.9: 16270276.0000 - val_fn
    0.9: 1606902.0000 - val_precision0.9: 0.8724 - val_recall0.9: 0.4919 - val_accuracy: 0.9091 - val_auc: 0.7866 - val_f1: 0.2772
    Epoch 2/20
    240/240 [==============================] - 123s 514ms/step - loss: 0.6145 - tp0.1: 10414504.0000 - fp0.1: 10347179.0000 - tn0.1: 56326272.0000 - fn0.1: 1555263.0000 - precision0.1: 0.5016
    - recall0.1: 0.8701 - tp0.3: 9433694.0000 - fp0.3: 4804465.0000 - tn0.3: 61868972.0000 - fn0.3: 2536073.0000 - precision0.3: 0.6626 - recall0.3: 0.7881 - tp0.5: 8253529.0000 - fp0.5: 21841
    43.0000 - tn0.5: 64489292.0000 - fn0.5: 3716238.0000 - precision0.5: 0.7907 - recall0.5: 0.6895 - tp0.7: 6893438.0000 - fp0.7: 964306.0000 - tn0.7: 65709120.0000 - fn0.7: 5076329.0000 - pr
    ecision0.7: 0.8773 - recall0.7: 0.5759 - tp0.9: 4825582.0000 - fp0.9: 253763.0000 - tn0.9: 66419652.0000 - fn0.9: 7144185.0000 - precision0.9: 0.9500 - recall0.9: 0.4031 - accuracy: 0.9250
     - auc: 0.9112 - f1: 0.2642 - val_loss: 0.7660 - val_tp0.1: 2600896.0000 - val_fp0.1: 1563848.0000 - val_tn0.1: 14934098.0000 - val_fn0.1: 561958.0000 - val_precision0.1: 0.6245 - val_reca
    ll0.1: 0.8223 - val_tp0.3: 2498170.0000 - val_fp0.3: 1105437.0000 - val_tn0.3: 15392509.0000 - val_fn0.3: 664684.0000 - val_precision0.3: 0.6932 - val_recall0.3: 0.7898 - val_tp0.5: 236736
    0.0000 - val_fp0.5: 720936.0000 - val_tn0.5: 15777010.0000 - val_fn0.5: 795494.0000 - val_precision0.5: 0.7666 - val_recall0.5: 0.7485 - val_tp0.7: 2247425.0000 - val_fp0.7: 512282.0000 -
    val_tn0.7: 15985664.0000 - val_fn0.7: 915429.0000 - val_precision0.7: 0.8144 - val_recall0.7: 0.7106 - val_tp0.9: 2064899.0000 - val_fp0.9: 341222.0000 - val_tn0.9: 16156724.0000 - val_fn0
    .9: 1097955.0000 - val_precision0.9: 0.8582 - val_recall0.9: 0.6529 - val_accuracy: 0.9229 - val_auc: 0.8919 - val_f1: 0.2772
    Epoch 3/20
    240/240 [==============================] - 123s 514ms/step - loss: 0.5296 - tp0.1: 10710507.0000 - fp0.1: 9627528.0000 - tn0.1: 57045904.0000 - fn0.1: 1259260.0000 - precision0.1: 0.5266 -
     recall0.1: 0.8948 - tp0.3: 9708967.0000 - fp0.3: 3890062.0000 - tn0.3: 62783372.0000 - fn0.3: 2260800.0000 - precision0.3: 0.7139 - recall0.3: 0.8111 - tp0.5: 8800137.0000 - fp0.5: 193587
    4.0000 - tn0.5: 64737520.0000 - fn0.5: 3169630.0000 - precision0.5: 0.8197 - recall0.5: 0.7352 - tp0.7: 7616621.0000 - fp0.7: 897975.0000 - tn0.7: 65775424.0000 - fn0.7: 4353146.0000 - pre
    cision0.7: 0.8945 - recall0.7: 0.6363 - tp0.9: 5653966.0000 - fp0.9: 272229.0000 - tn0.9: 66401216.0000 - fn0.9: 6315801.0000 - precision0.9: 0.9541 - recall0.9: 0.4724 - accuracy: 0.9351
    - auc: 0.9281 - f1: 0.2642 - val_loss: 0.6681 - val_tp0.1: 2593120.0000 - val_fp0.1: 1019904.0000 - val_tn0.1: 15478042.0000 - val_fn0.1: 569734.0000 - val_precision0.1: 0.7177 - val_recal
    l0.1: 0.8199 - val_tp0.3: 2244784.0000 - val_fp0.3: 338835.0000 - val_tn0.3: 16159111.0000 - val_fn0.3: 918070.0000 - val_precision0.3: 0.8689 - val_recall0.3: 0.7097 - val_tp0.5: 2014697.
    0000 - val_fp0.5: 180709.0000 - val_tn0.5: 16317237.0000 - val_fn0.5: 1148157.0000 - val_precision0.5: 0.9177 - val_recall0.5: 0.6370 - val_tp0.7: 1651445.0000 - val_fp0.7: 66392.0000 - va
    l_tn0.7: 16431554.0000 - val_fn0.7: 1511409.0000 - val_precision0.7: 0.9614 - val_recall0.7: 0.5221 - val_tp0.9: 1122584.0000 - val_fp0.9: 10626.0000 - val_tn0.9: 16487320.0000 - val_fn0.9
    : 2040270.0000 - val_precision0.9: 0.9906 - val_recall0.9: 0.3549 - val_accuracy: 0.9324 - val_auc: 0.9151 - val_f1: 0.2772
    Epoch 4/20
    240/240 [==============================] - 123s 514ms/step - loss: 0.4720 - tp0.1: 10877075.0000 - fp0.1: 8671019.0000 - tn0.1: 58002392.0000 - fn0.1: 1092692.0000 - precision0.1: 0.5564 -
     recall0.1: 0.9087 - tp0.3: 9936344.0000 - fp0.3: 3411676.0000 - tn0.3: 63261736.0000 - fn0.3: 2033423.0000 - precision0.3: 0.7444 - recall0.3: 0.8301 - tp0.5: 9211814.0000 - fp0.5: 187395
    8.0000 - tn0.5: 64799456.0000 - fn0.5: 2757953.0000 - precision0.5: 0.8310 - recall0.5: 0.7696 - tp0.7: 8139780.0000 - fp0.7: 898839.0000 - tn0.7: 65774596.0000 - fn0.7: 3829987.0000 - pre
    cision0.7: 0.9006 - recall0.7: 0.6800 - tp0.9: 6238056.0000 - fp0.9: 255726.0000 - tn0.9: 66417696.0000 - fn0.9: 5731711.0000 - precision0.9: 0.9606 - recall0.9: 0.5212 - accuracy: 0.9411
    - auc: 0.9384 - f1: 0.2642 - val_loss: 0.4134 - val_tp0.1: 3012908.0000 - val_fp0.1: 2692890.0000 - val_tn0.1: 13805056.0000 - val_fn0.1: 149946.0000 - val_precision0.1: 0.5280 - val_recal
    l0.1: 0.9526 - val_tp0.3: 2844859.0000 - val_fp0.3: 1085275.0000 - val_tn0.3: 15412671.0000 - val_fn0.3: 317995.0000 - val_precision0.3: 0.7239 - val_recall0.3: 0.8995 - val_tp0.5: 2702953
    .0000 - val_fp0.5: 596686.0000 - val_tn0.5: 15901260.0000 - val_fn0.5: 459901.0000 - val_precision0.5: 0.8192 - val_recall0.5: 0.8546 - val_tp0.7: 2523224.0000 - val_fp0.7: 328369.0000 - v
    al_tn0.7: 16169577.0000 - val_fn0.7: 639630.0000 - val_precision0.7: 0.8848 - val_recall0.7: 0.7978 - val_tp0.9: 2192428.0000 - val_fp0.9: 138765.0000 - val_tn0.9: 16359181.0000 - val_fn0.
    9: 970426.0000 - val_precision0.9: 0.9405 - val_recall0.9: 0.6932 - val_accuracy: 0.9463 - val_auc: 0.9625 - val_f1: 0.2772
    Epoch 5/20
    240/240 [==============================] - 124s 515ms/step - loss: 0.4310 - tp0.1: 10975989.0000 - fp0.1: 7934457.0000 - tn0.1: 58738992.0000 - fn0.1: 993778.0000 - precision0.1: 0.5804 -
    recall0.1: 0.9170 - tp0.3: 10114842.0000 - fp0.3: 3195737.0000 - tn0.3: 63477704.0000 - fn0.3: 1854925.0000 - precision0.3: 0.7599 - recall0.3: 0.8450 - tp0.5: 9339053.0000 - fp0.5: 161448
    5.0000 - tn0.5: 65058948.0000 - fn0.5: 2630714.0000 - precision0.5: 0.8526 - recall0.5: 0.7802 - tp0.7: 8449558.0000 - fp0.7: 805407.0000 - tn0.7: 65868032.0000 - fn0.7: 3520209.0000 - pre
    cision0.7: 0.9130 - recall0.7: 0.7059 - tp0.9: 6730077.0000 - fp0.9: 232993.0000 - tn0.9: 66440428.0000 - fn0.9: 5239690.0000 - precision0.9: 0.9665 - recall0.9: 0.5623 - accuracy: 0.9460
    - auc: 0.9454 - f1: 0.2642 - val_loss: 0.4139 - val_tp0.1: 3048945.0000 - val_fp0.1: 2758234.0000 - val_tn0.1: 13739712.0000 - val_fn0.1: 113909.0000 - val_precision0.1: 0.5250 - val_recal
    l0.1: 0.9640 - val_tp0.3: 2947363.0000 - val_fp0.3: 1444749.0000 - val_tn0.3: 15053197.0000 - val_fn0.3: 215491.0000 - val_precision0.3: 0.6711 - val_recall0.3: 0.9319 - val_tp0.5: 2843052
    .0000 - val_fp0.5: 901333.0000 - val_tn0.5: 15596613.0000 - val_fn0.5: 319802.0000 - val_precision0.5: 0.7593 - val_recall0.5: 0.8989 - val_tp0.7: 2732073.0000 - val_fp0.7: 569323.0000 - v
    al_tn0.7: 15928623.0000 - val_fn0.7: 430781.0000 - val_precision0.7: 0.8276 - val_recall0.7: 0.8638 - val_tp0.9: 2540735.0000 - val_fp0.9: 283228.0000 - val_tn0.9: 16214718.0000 - val_fn0.
    9: 622119.0000 - val_precision0.9: 0.8997 - val_recall0.9: 0.8033 - val_accuracy: 0.9379 - val_auc: 0.9680 - val_f1: 0.2772
    Epoch 6/20
    240/240 [==============================] - 123s 514ms/step - loss: 0.4055 - tp0.1: 11005938.0000 - fp0.1: 7117253.0000 - tn0.1: 59556168.0000 - fn0.1: 963829.0000 - precision0.1: 0.6073 -
    recall0.1: 0.9195 - tp0.3: 10252977.0000 - fp0.3: 2993472.0000 - tn0.3: 63679960.0000 - fn0.3: 1716790.0000 - precision0.3: 0.7740 - recall0.3: 0.8566 - tp0.5: 9600067.0000 - fp0.5: 159785
    2.0000 - tn0.5: 65075568.0000 - fn0.5: 2369700.0000 - precision0.5: 0.8573 - recall0.5: 0.8020 - tp0.7: 8735547.0000 - fp0.7: 775018.0000 - tn0.7: 65898436.0000 - fn0.7: 3234220.0000 - pre
    cision0.7: 0.9185 - recall0.7: 0.7298 - tp0.9: 7088496.0000 - fp0.9: 233616.0000 - tn0.9: 66439832.0000 - fn0.9: 4881271.0000 - precision0.9: 0.9681 - recall0.9: 0.5922 - accuracy: 0.9496
    - auc: 0.9484 - f1: 0.2642 - val_loss: 0.3925 - val_tp0.1: 3016157.0000 - val_fp0.1: 2494175.0000 - val_tn0.1: 14003771.0000 - val_fn0.1: 146697.0000 - val_precision0.1: 0.5474 - val_recal
    l0.1: 0.9536 - val_tp0.3: 2913629.0000 - val_fp0.3: 1206347.0000 - val_tn0.3: 15291599.0000 - val_fn0.3: 249225.0000 - val_precision0.3: 0.7072 - val_recall0.3: 0.9212 - val_tp0.5: 2794468
    .0000 - val_fp0.5: 707843.0000 - val_tn0.5: 15790103.0000 - val_fn0.5: 368386.0000 - val_precision0.5: 0.7979 - val_recall0.5: 0.8835 - val_tp0.7: 2631745.0000 - val_fp0.7: 409739.0000 - v
    al_tn0.7: 16088207.0000 - val_fn0.7: 531109.0000 - val_precision0.7: 0.8653 - val_recall0.7: 0.8321 - val_tp0.9: 2298554.0000 - val_fp0.9: 168326.0000 - val_tn0.9: 16329620.0000 - val_fn0.
    9: 864300.0000 - val_precision0.9: 0.9318 - val_recall0.9: 0.7267 - val_accuracy: 0.9453 - val_auc: 0.9633 - val_f1: 0.2772
    Epoch 7/20
    240/240 [==============================] - 123s 515ms/step - loss: 0.3854 - tp0.1: 11113311.0000 - fp0.1: 7284589.0000 - tn0.1: 59388840.0000 - fn0.1: 856456.0000 - precision0.1: 0.6041 -
    recall0.1: 0.9284 - tp0.3: 10356441.0000 - fp0.3: 3026811.0000 - tn0.3: 63646640.0000 - fn0.3: 1613326.0000 - precision0.3: 0.7738 - recall0.3: 0.8652 - tp0.5: 9718595.0000 - fp0.5: 163300
    4.0000 - tn0.5: 65040420.0000 - fn0.5: 2251172.0000 - precision0.5: 0.8561 - recall0.5: 0.8119 - tp0.7: 8835263.0000 - fp0.7: 784778.0000 - tn0.7: 65888660.0000 - fn0.7: 3134504.0000 - pre
    cision0.7: 0.9184 - recall0.7: 0.7381 - tp0.9: 7228050.0000 - fp0.9: 242511.0000 - tn0.9: 66430912.0000 - fn0.9: 4741717.0000 - precision0.9: 0.9675 - recall0.9: 0.6039 - accuracy: 0.9506
    - auc: 0.9535 - f1: 0.2642 - val_loss: 0.4443 - val_tp0.1: 2906183.0000 - val_fp0.1: 1626644.0000 - val_tn0.1: 14871302.0000 - val_fn0.1: 256671.0000 - val_precision0.1: 0.6411 - val_recal
    l0.1: 0.9188 - val_tp0.3: 2816271.0000 - val_fp0.3: 1160666.0000 - val_tn0.3: 15337280.0000 - val_fn0.3: 346583.0000 - val_precision0.3: 0.7082 - val_recall0.3: 0.8904 - val_tp0.5: 2689144
    .0000 - val_fp0.5: 809812.0000 - val_tn0.5: 15688134.0000 - val_fn0.5: 473710.0000 - val_precision0.5: 0.7686 - val_recall0.5: 0.8502 - val_tp0.7: 2488156.0000 - val_fp0.7: 481300.0000 - v
    al_tn0.7: 16016646.0000 - val_fn0.7: 674698.0000 - val_precision0.7: 0.8379 - val_recall0.7: 0.7867 - val_tp0.9: 2127831.0000 - val_fp0.9: 230399.0000 - val_tn0.9: 16267547.0000 - val_fn0.
    9: 1035023.0000 - val_precision0.9: 0.9023 - val_recall0.9: 0.6728 - val_accuracy: 0.9347 - val_auc: 0.9452 - val_f1: 0.2772
    Epoch 8/20
    240/240 [==============================] - 124s 515ms/step - loss: 0.3596 - tp0.1: 11147835.0000 - fp0.1: 6521359.0000 - tn0.1: 60152048.0000 - fn0.1: 821932.0000 - precision0.1: 0.6309 -
    recall0.1: 0.9313 - tp0.3: 10458403.0000 - fp0.3: 2774423.0000 - tn0.3: 63899012.0000 - fn0.3: 1511364.0000 - precision0.3: 0.7903 - recall0.3: 0.8737 - tp0.5: 9862220.0000 - fp0.5: 151598
    7.0000 - tn0.5: 65157436.0000 - fn0.5: 2107547.0000 - precision0.5: 0.8668 - recall0.5: 0.8239 - tp0.7: 9063431.0000 - fp0.7: 749399.0000 - tn0.7: 65924016.0000 - fn0.7: 2906336.0000 - pre
    cision0.7: 0.9236 - recall0.7: 0.7572 - tp0.9: 7504772.0000 - fp0.9: 226379.0000 - tn0.9: 66447060.0000 - fn0.9: 4464995.0000 - precision0.9: 0.9707 - recall0.9: 0.6270 - accuracy: 0.9539
    - auc: 0.9563 - f1: 0.2642 - val_loss: 0.5498 - val_tp0.1: 3055124.0000 - val_fp0.1: 4118468.0000 - val_tn0.1: 12379478.0000 - val_fn0.1: 107730.0000 - val_precision0.1: 0.4259 - val_recal
    l0.1: 0.9659 - val_tp0.3: 2939354.0000 - val_fp0.3: 2257326.0000 - val_tn0.3: 14240620.0000 - val_fn0.3: 223500.0000 - val_precision0.3: 0.5656 - val_recall0.3: 0.9293 - val_tp0.5: 2829932
    .0000 - val_fp0.5: 1526939.0000 - val_tn0.5: 14971007.0000 - val_fn0.5: 332922.0000 - val_precision0.5: 0.6495 - val_recall0.5: 0.8947 - val_tp0.7: 2642371.0000 - val_fp0.7: 901220.0000 -
    val_tn0.7: 15596726.0000 - val_fn0.7: 520483.0000 - val_precision0.7: 0.7457 - val_recall0.7: 0.8354 - val_tp0.9: 2236370.0000 - val_fp0.9: 400953.0000 - val_tn0.9: 16096993.0000 - val_fn0
    .9: 926484.0000 - val_precision0.9: 0.8480 - val_recall0.9: 0.7071 - val_accuracy: 0.9054 - val_auc: 0.9561 - val_f1: 0.2772
    Epoch 9/20
    240/240 [==============================] - 123s 514ms/step - loss: 0.3587 - tp0.1: 11154292.0000 - fp0.1: 6547650.0000 - tn0.1: 60125796.0000 - fn0.1: 815475.0000 - precision0.1: 0.6301 -
    recall0.1: 0.9319 - tp0.3: 10488369.0000 - fp0.3: 2868809.0000 - tn0.3: 63804604.0000 - fn0.3: 1481398.0000 - precision0.3: 0.7852 - recall0.3: 0.8762 - tp0.5: 9875989.0000 - fp0.5: 153301
    6.0000 - tn0.5: 65140432.0000 - fn0.5: 2093778.0000 - precision0.5: 0.8656 - recall0.5: 0.8251 - tp0.7: 9098782.0000 - fp0.7: 781237.0000 - tn0.7: 65892200.0000 - fn0.7: 2870985.0000 - pre
    cision0.7: 0.9209 - recall0.7: 0.7601 - tp0.9: 7462273.0000 - fp0.9: 220634.0000 - tn0.9: 66452784.0000 - fn0.9: 4507494.0000 - precision0.9: 0.9713 - recall0.9: 0.6234 - accuracy: 0.9539
    - auc: 0.9568 - f1: 0.2642 - val_loss: 0.3098 - val_tp0.1: 3044237.0000 - val_fp0.1: 1845158.0000 - val_tn0.1: 14652788.0000 - val_fn0.1: 118617.0000 - val_precision0.1: 0.6226 - val_recal
    l0.1: 0.9625 - val_tp0.3: 2952931.0000 - val_fp0.3: 1010870.0000 - val_tn0.3: 15487076.0000 - val_fn0.3: 209923.0000 - val_precision0.3: 0.7450 - val_recall0.3: 0.9336 - val_tp0.5: 2818007
    .0000 - val_fp0.5: 560464.0000 - val_tn0.5: 15937482.0000 - val_fn0.5: 344847.0000 - val_precision0.5: 0.8341 - val_recall0.5: 0.8910 - val_tp0.7: 2675497.0000 - val_fp0.7: 330511.0000 - v
    al_tn0.7: 16167435.0000 - val_fn0.7: 487357.0000 - val_precision0.7: 0.8900 - val_recall0.7: 0.8459 - val_tp0.9: 2356811.0000 - val_fp0.9: 121488.0000 - val_tn0.9: 16376458.0000 - val_fn0.
    9: 806043.0000 - val_precision0.9: 0.9510 - val_recall0.9: 0.7452 - val_accuracy: 0.9540 - val_auc: 0.9720 - val_f1: 0.2772
    Epoch 10/20
    240/240 [==============================] - 123s 514ms/step - loss: 0.3310 - tp0.1: 11220800.0000 - fp0.1: 6024418.0000 - tn0.1: 60649028.0000 - fn0.1: 748967.0000 - precision0.1: 0.6507 -
    recall0.1: 0.9374 - tp0.3: 10602011.0000 - fp0.3: 2619415.0000 - tn0.3: 64054032.0000 - fn0.3: 1367756.0000 - precision0.3: 0.8019 - recall0.3: 0.8857 - tp0.5: 10035760.0000 - fp0.5: 14108
    57.0000 - tn0.5: 65262568.0000 - fn0.5: 1934007.0000 - precision0.5: 0.8767 - recall0.5: 0.8384 - tp0.7: 9310399.0000 - fp0.7: 714073.0000 - tn0.7: 65959376.0000 - fn0.7: 2659368.0000 - pr
    ecision0.7: 0.9288 - recall0.7: 0.7778 - tp0.9: 7802441.0000 - fp0.9: 197965.0000 - tn0.9: 66475472.0000 - fn0.9: 4167326.0000 - precision0.9: 0.9753 - recall0.9: 0.6518 - accuracy: 0.9575
     - auc: 0.9608 - f1: 0.2642 - val_loss: 0.3701 - val_tp0.1: 3063278.0000 - val_fp0.1: 2656513.0000 - val_tn0.1: 13841433.0000 - val_fn0.1: 99576.0000 - val_precision0.1: 0.5356 - val_recal
    l0.1: 0.9685 - val_tp0.3: 2959133.0000 - val_fp0.3: 1261902.0000 - val_tn0.3: 15236044.0000 - val_fn0.3: 203721.0000 - val_precision0.3: 0.7010 - val_recall0.3: 0.9356 - val_tp0.5: 2866997
    .0000 - val_fp0.5: 802391.0000 - val_tn0.5: 15695555.0000 - val_fn0.5: 295857.0000 - val_precision0.5: 0.7813 - val_recall0.5: 0.9065 - val_tp0.7: 2730525.0000 - val_fp0.7: 462170.0000 - v
    al_tn0.7: 16035776.0000 - val_fn0.7: 432329.0000 - val_precision0.7: 0.8552 - val_recall0.7: 0.8633 - val_tp0.9: 2464570.0000 - val_fp0.9: 189700.0000 - val_tn0.9: 16308246.0000 - val_fn0.
    9: 698284.0000 - val_precision0.9: 0.9285 - val_recall0.9: 0.7792 - val_accuracy: 0.9441 - val_auc: 0.9728 - val_f1: 0.2772
    Epoch 11/20
    240/240 [==============================] - 123s 514ms/step - loss: 0.3233 - tp0.1: 11231645.0000 - fp0.1: 5931503.0000 - tn0.1: 60741932.0000 - fn0.1: 738122.0000 - precision0.1: 0.6544 -
    recall0.1: 0.9383 - tp0.3: 10598514.0000 - fp0.3: 2444604.0000 - tn0.3: 64228848.0000 - fn0.3: 1371253.0000 - precision0.3: 0.8126 - recall0.3: 0.8854 - tp0.5: 10075292.0000 - fp0.5: 13321
    40.0000 - tn0.5: 65341256.0000 - fn0.5: 1894475.0000 - precision0.5: 0.8832 - recall0.5: 0.8417 - tp0.7: 9378243.0000 - fp0.7: 674133.0000 - tn0.7: 65999280.0000 - fn0.7: 2591524.0000 - pr
    ecision0.7: 0.9329 - recall0.7: 0.7835 - tp0.9: 7966249.0000 - fp0.9: 193209.0000 - tn0.9: 66480236.0000 - fn0.9: 4003518.0000 - precision0.9: 0.9763 - recall0.9: 0.6655 - accuracy: 0.9590
     - auc: 0.9619 - f1: 0.2642 - val_loss: 0.2986 - val_tp0.1: 3049223.0000 - val_fp0.1: 1879104.0000 - val_tn0.1: 14618842.0000 - val_fn0.1: 113631.0000 - val_precision0.1: 0.6187 - val_reca
    ll0.1: 0.9641 - val_tp0.3: 2957119.0000 - val_fp0.3: 970547.0000 - val_tn0.3: 15527399.0000 - val_fn0.3: 205735.0000 - val_precision0.3: 0.7529 - val_recall0.3: 0.9350 - val_tp0.5: 2848369
    .0000 - val_fp0.5: 566365.0000 - val_tn0.5: 15931581.0000 - val_fn0.5: 314485.0000 - val_precision0.5: 0.8341 - val_recall0.5: 0.9006 - val_tp0.7: 2684500.0000 - val_fp0.7: 298668.0000 - v
    al_tn0.7: 16199278.0000 - val_fn0.7: 478354.0000 - val_precision0.7: 0.8999 - val_recall0.7: 0.8488 - val_tp0.9: 2264810.0000 - val_fp0.9: 80756.0000 - val_tn0.9: 16417190.0000 - val_fn0.9
    : 898044.0000 - val_precision0.9: 0.9656 - val_recall0.9: 0.7161 - val_accuracy: 0.9552 - val_auc: 0.9738 - val_f1: 0.2772
    Epoch 12/20
    240/240 [==============================] - 123s 514ms/step - loss: 0.3085 - tp0.1: 11270470.0000 - fp0.1: 5619369.0000 - tn0.1: 61054068.0000 - fn0.1: 699297.0000 - precision0.1: 0.6673 -
    recall0.1: 0.9416 - tp0.3: 10714167.0000 - fp0.3: 2487268.0000 - tn0.3: 64186176.0000 - fn0.3: 1255600.0000 - precision0.3: 0.8116 - recall0.3: 0.8951 - tp0.5: 10175108.0000 - fp0.5: 13504
    36.0000 - tn0.5: 65323020.0000 - fn0.5: 1794659.0000 - precision0.5: 0.8828 - recall0.5: 0.8501 - tp0.7: 9471869.0000 - fp0.7: 668828.0000 - tn0.7: 66004600.0000 - fn0.7: 2497898.0000 - pr
    ecision0.7: 0.9340 - recall0.7: 0.7913 - tp0.9: 8087818.0000 - fp0.9: 195376.0000 - tn0.9: 66478052.0000 - fn0.9: 3881949.0000 - precision0.9: 0.9764 - recall0.9: 0.6757 - accuracy: 0.9600
     - auc: 0.9640 - f1: 0.2642 - val_loss: 0.3350 - val_tp0.1: 3080571.0000 - val_fp0.1: 2712529.0000 - val_tn0.1: 13785417.0000 - val_fn0.1: 82283.0000 - val_precision0.1: 0.5318 - val_recal
    l0.1: 0.9740 - val_tp0.3: 2997438.0000 - val_fp0.3: 1233176.0000 - val_tn0.3: 15264770.0000 - val_fn0.3: 165416.0000 - val_precision0.3: 0.7085 - val_recall0.3: 0.9477 - val_tp0.5: 2901164
    .0000 - val_fp0.5: 678226.0000 - val_tn0.5: 15819720.0000 - val_fn0.5: 261690.0000 - val_precision0.5: 0.8105 - val_recall0.5: 0.9173 - val_tp0.7: 2761496.0000 - val_fp0.7: 360082.0000 - v
    al_tn0.7: 16137864.0000 - val_fn0.7: 401358.0000 - val_precision0.7: 0.8846 - val_recall0.7: 0.8731 - val_tp0.9: 2425682.0000 - val_fp0.9: 106600.0000 - val_tn0.9: 16391346.0000 - val_fn0.
    9: 737172.0000 - val_precision0.9: 0.9579 - val_recall0.9: 0.7669 - val_accuracy: 0.9522 - val_auc: 0.9775 - val_f1: 0.2772
    Epoch 13/20
    240/240 [==============================] - 123s 513ms/step - loss: 0.2998 - tp0.1: 11289427.0000 - fp0.1: 5435141.0000 - tn0.1: 61238288.0000 - fn0.1: 680340.0000 - precision0.1: 0.6750 -
    recall0.1: 0.9432 - tp0.3: 10775991.0000 - fp0.3: 2541120.0000 - tn0.3: 64132324.0000 - fn0.3: 1193776.0000 - precision0.3: 0.8092 - recall0.3: 0.9003 - tp0.5: 10236179.0000 - fp0.5: 13511
    65.0000 - tn0.5: 65322272.0000 - fn0.5: 1733588.0000 - precision0.5: 0.8834 - recall0.5: 0.8552 - tp0.7: 9552300.0000 - fp0.7: 667311.0000 - tn0.7: 66006116.0000 - fn0.7: 2417467.0000 - pr
    ecision0.7: 0.9347 - recall0.7: 0.7980 - tp0.9: 8197249.0000 - fp0.9: 192011.0000 - tn0.9: 66481424.0000 - fn0.9: 3772518.0000 - precision0.9: 0.9771 - recall0.9: 0.6848 - accuracy: 0.9608
     - auc: 0.9652 - f1: 0.2642 - val_loss: 0.2859 - val_tp0.1: 2992948.0000 - val_fp0.1: 1104053.0000 - val_tn0.1: 15393893.0000 - val_fn0.1: 169906.0000 - val_precision0.1: 0.7305 - val_reca
    ll0.1: 0.9463 - val_tp0.3: 2859768.0000 - val_fp0.3: 511820.0000 - val_tn0.3: 15986126.0000 - val_fn0.3: 303086.0000 - val_precision0.3: 0.8482 - val_recall0.3: 0.9042 - val_tp0.5: 2679052
    .0000 - val_fp0.5: 241764.0000 - val_tn0.5: 16256182.0000 - val_fn0.5: 483802.0000 - val_precision0.5: 0.9172 - val_recall0.5: 0.8470 - val_tp0.7: 2445930.0000 - val_fp0.7: 113353.0000 - v
    al_tn0.7: 16384593.0000 - val_fn0.7: 716924.0000 - val_precision0.7: 0.9557 - val_recall0.7: 0.7733 - val_tp0.9: 2024394.0000 - val_fp0.9: 30607.0000 - val_tn0.9: 16467339.0000 - val_fn0.9
    : 1138460.0000 - val_precision0.9: 0.9851 - val_recall0.9: 0.6401 - val_accuracy: 0.9631 - val_auc: 0.9677 - val_f1: 0.2772
    Epoch 14/20
    240/240 [==============================] - 123s 513ms/step - loss: 0.2873 - tp0.1: 11325492.0000 - fp0.1: 5158653.0000 - tn0.1: 61514784.0000 - fn0.1: 644275.0000 - precision0.1: 0.6871 -
    recall0.1: 0.9462 - tp0.3: 10839018.0000 - fp0.3: 2435459.0000 - tn0.3: 64237968.0000 - fn0.3: 1130749.0000 - precision0.3: 0.8165 - recall0.3: 0.9055 - tp0.5: 10316818.0000 - fp0.5: 13405
    37.0000 - tn0.5: 65332924.0000 - fn0.5: 1652949.0000 - precision0.5: 0.8850 - recall0.5: 0.8619 - tp0.7: 9615679.0000 - fp0.7: 663572.0000 - tn0.7: 66009872.0000 - fn0.7: 2354088.0000 - pr
    ecision0.7: 0.9354 - recall0.7: 0.8033 - tp0.9: 8226896.0000 - fp0.9: 190675.0000 - tn0.9: 66482748.0000 - fn0.9: 3742871.0000 - precision0.9: 0.9773 - recall0.9: 0.6873 - accuracy: 0.9619
     - auc: 0.9668 - f1: 0.2642 - val_loss: 0.2843 - val_tp0.1: 3018528.0000 - val_fp0.1: 1341368.0000 - val_tn0.1: 15156578.0000 - val_fn0.1: 144326.0000 - val_precision0.1: 0.6923 - val_reca
    ll0.1: 0.9544 - val_tp0.3: 2884535.0000 - val_fp0.3: 604165.0000 - val_tn0.3: 15893781.0000 - val_fn0.3: 278319.0000 - val_precision0.3: 0.8268 - val_recall0.3: 0.9120 - val_tp0.5: 2750904
    .0000 - val_fp0.5: 350090.0000 - val_tn0.5: 16147856.0000 - val_fn0.5: 411950.0000 - val_precision0.5: 0.8871 - val_recall0.5: 0.8698 - val_tp0.7: 2579988.0000 - val_fp0.7: 191060.0000 - v
    al_tn0.7: 16306886.0000 - val_fn0.7: 582866.0000 - val_precision0.7: 0.9311 - val_recall0.7: 0.8157 - val_tp0.9: 2263427.0000 - val_fp0.9: 74463.0000 - val_tn0.9: 16423483.0000 - val_fn0.9
    : 899427.0000 - val_precision0.9: 0.9681 - val_recall0.9: 0.7156 - val_accuracy: 0.9612 - val_auc: 0.9707 - val_f1: 0.2772
    Epoch 15/20
    240/240 [==============================] - 123s 513ms/step - loss: 0.2841 - tp0.1: 11339440.0000 - fp0.1: 5281029.0000 - tn0.1: 61392416.0000 - fn0.1: 630327.0000 - precision0.1: 0.6823 -
    recall0.1: 0.9473 - tp0.3: 10822522.0000 - fp0.3: 2360182.0000 - tn0.3: 64313252.0000 - fn0.3: 1147245.0000 - precision0.3: 0.8210 - recall0.3: 0.9042 - tp0.5: 10312399.0000 - fp0.5: 12761
    78.0000 - tn0.5: 65397248.0000 - fn0.5: 1657368.0000 - precision0.5: 0.8899 - recall0.5: 0.8615 - tp0.7: 9644338.0000 - fp0.7: 625693.0000 - tn0.7: 66047756.0000 - fn0.7: 2325429.0000 - pr
    ecision0.7: 0.9391 - recall0.7: 0.8057 - tp0.9: 8361871.0000 - fp0.9: 174964.0000 - tn0.9: 66498432.0000 - fn0.9: 3607896.0000 - precision0.9: 0.9795 - recall0.9: 0.6986 - accuracy: 0.9627
     - auc: 0.9678 - f1: 0.2642 - val_loss: 0.2785 - val_tp0.1: 3071530.0000 - val_fp0.1: 1940363.0000 - val_tn0.1: 14557583.0000 - val_fn0.1: 91324.0000 - val_precision0.1: 0.6128 - val_recal
    l0.1: 0.9711 - val_tp0.3: 2963120.0000 - val_fp0.3: 829784.0000 - val_tn0.3: 15668162.0000 - val_fn0.3: 199734.0000 - val_precision0.3: 0.7812 - val_recall0.3: 0.9369 - val_tp0.5: 2835343.
    0000 - val_fp0.5: 408144.0000 - val_tn0.5: 16089802.0000 - val_fn0.5: 327511.0000 - val_precision0.5: 0.8742 - val_recall0.5: 0.8965 - val_tp0.7: 2617565.0000 - val_fp0.7: 178411.0000 - va
    l_tn0.7: 16319535.0000 - val_fn0.7: 545289.0000 - val_precision0.7: 0.9362 - val_recall0.7: 0.8276 - val_tp0.9: 2212166.0000 - val_fp0.9: 48362.0000 - val_tn0.9: 16449584.0000 - val_fn0.9:
     950688.0000 - val_precision0.9: 0.9786 - val_recall0.9: 0.6994 - val_accuracy: 0.9626 - val_auc: 0.9783 - val_f1: 0.2772
    Epoch 16/20
    240/240 [==============================] - 123s 513ms/step - loss: 0.2733 - tp0.1: 11349878.0000 - fp0.1: 4939772.0000 - tn0.1: 61733644.0000 - fn0.1: 619889.0000 - precision0.1: 0.6968 -
    recall0.1: 0.9482 - tp0.3: 10872946.0000 - fp0.3: 2277823.0000 - tn0.3: 64395616.0000 - fn0.3: 1096821.0000 - precision0.3: 0.8268 - recall0.3: 0.9084 - tp0.5: 10390972.0000 - fp0.5: 12472
    64.0000 - tn0.5: 65426192.0000 - fn0.5: 1578795.0000 - precision0.5: 0.8928 - recall0.5: 0.8681 - tp0.7: 9736581.0000 - fp0.7: 597016.0000 - tn0.7: 66076408.0000 - fn0.7: 2233186.0000 - pr
    ecision0.7: 0.9422 - recall0.7: 0.8134 - tp0.9: 8491483.0000 - fp0.9: 164275.0000 - tn0.9: 66509160.0000 - fn0.9: 3478284.0000 - precision0.9: 0.9810 - recall0.9: 0.7094 - accuracy: 0.9641
     - auc: 0.9686 - f1: 0.2642 - val_loss: 0.2719 - val_tp0.1: 3021839.0000 - val_fp0.1: 1339815.0000 - val_tn0.1: 15158131.0000 - val_fn0.1: 141015.0000 - val_precision0.1: 0.6928 - val_reca
    ll0.1: 0.9554 - val_tp0.3: 2908072.0000 - val_fp0.3: 621879.0000 - val_tn0.3: 15876067.0000 - val_fn0.3: 254782.0000 - val_precision0.3: 0.8238 - val_recall0.3: 0.9194 - val_tp0.5: 2802958
    .0000 - val_fp0.5: 363679.0000 - val_tn0.5: 16134267.0000 - val_fn0.5: 359896.0000 - val_precision0.5: 0.8852 - val_recall0.5: 0.8862 - val_tp0.7: 2647510.0000 - val_fp0.7: 188747.0000 - v
    al_tn0.7: 16309199.0000 - val_fn0.7: 515344.0000 - val_precision0.7: 0.9335 - val_recall0.7: 0.8371 - val_tp0.9: 2292567.0000 - val_fp0.9: 53401.0000 - val_tn0.9: 16444545.0000 - val_fn0.9
    : 870287.0000 - val_precision0.9: 0.9772 - val_recall0.9: 0.7248 - val_accuracy: 0.9632 - val_auc: 0.9717 - val_f1: 0.2772
    Epoch 17/20
    240/240 [==============================] - 123s 512ms/step - loss: 0.2657 - tp0.1: 11376204.0000 - fp0.1: 4892198.0000 - tn0.1: 61781224.0000 - fn0.1: 593563.0000 - precision0.1: 0.6993 -
    recall0.1: 0.9504 - tp0.3: 10908750.0000 - fp0.3: 2245727.0000 - tn0.3: 64427696.0000 - fn0.3: 1061017.0000 - precision0.3: 0.8293 - recall0.3: 0.9114 - tp0.5: 10433698.0000 - fp0.5: 12316
    34.0000 - tn0.5: 65441800.0000 - fn0.5: 1536069.0000 - precision0.5: 0.8944 - recall0.5: 0.8717 - tp0.7: 9778950.0000 - fp0.7: 598659.0000 - tn0.7: 66074788.0000 - fn0.7: 2190817.0000 - pr
    ecision0.7: 0.9423 - recall0.7: 0.8170 - tp0.9: 8517790.0000 - fp0.9: 162128.0000 - tn0.9: 66511320.0000 - fn0.9: 3451977.0000 - precision0.9: 0.9813 - recall0.9: 0.7116 - accuracy: 0.9648
     - auc: 0.9699 - f1: 0.2642 - val_loss: 0.2613 - val_tp0.1: 3051492.0000 - val_fp0.1: 1505492.0000 - val_tn0.1: 14992454.0000 - val_fn0.1: 111362.0000 - val_precision0.1: 0.6696 - val_reca
    ll0.1: 0.9648 - val_tp0.3: 2939774.0000 - val_fp0.3: 659951.0000 - val_tn0.3: 15837995.0000 - val_fn0.3: 223080.0000 - val_precision0.3: 0.8167 - val_recall0.3: 0.9295 - val_tp0.5: 2833524
    .0000 - val_fp0.5: 395235.0000 - val_tn0.5: 16102711.0000 - val_fn0.5: 329330.0000 - val_precision0.5: 0.8776 - val_recall0.5: 0.8959 - val_tp0.7: 2680855.0000 - val_fp0.7: 215919.0000 - v
    al_tn0.7: 16282027.0000 - val_fn0.7: 481999.0000 - val_precision0.7: 0.9255 - val_recall0.7: 0.8476 - val_tp0.9: 2359116.0000 - val_fp0.9: 75353.0000 - val_tn0.9: 16422593.0000 - val_fn0.9
    : 803738.0000 - val_precision0.9: 0.9690 - val_recall0.9: 0.7459 - val_accuracy: 0.9631 - val_auc: 0.9765 - val_f1: 0.2772
    Epoch 18/20
    240/240 [==============================] - 123s 513ms/step - loss: 0.2598 - tp0.1: 11386263.0000 - fp0.1: 4766580.0000 - tn0.1: 61906832.0000 - fn0.1: 583504.0000 - precision0.1: 0.7049 -
    recall0.1: 0.9513 - tp0.3: 10931522.0000 - fp0.3: 2185962.0000 - tn0.3: 64487472.0000 - fn0.3: 1038245.0000 - precision0.3: 0.8334 - recall0.3: 0.9133 - tp0.5: 10479377.0000 - fp0.5: 12310
    07.0000 - tn0.5: 65442416.0000 - fn0.5: 1490390.0000 - precision0.5: 0.8949 - recall0.5: 0.8755 - tp0.7: 9829899.0000 - fp0.7: 592904.0000 - tn0.7: 66080536.0000 - fn0.7: 2139868.0000 - pr
    ecision0.7: 0.9431 - recall0.7: 0.8212 - tp0.9: 8602261.0000 - fp0.9: 161222.0000 - tn0.9: 66512188.0000 - fn0.9: 3367506.0000 - precision0.9: 0.9816 - recall0.9: 0.7187 - accuracy: 0.9654
     - auc: 0.9706 - f1: 0.2642 - val_loss: 0.2608 - val_tp0.1: 3020760.0000 - val_fp0.1: 1219927.0000 - val_tn0.1: 15278019.0000 - val_fn0.1: 142094.0000 - val_precision0.1: 0.7123 - val_reca
    ll0.1: 0.9551 - val_tp0.3: 2909302.0000 - val_fp0.3: 580392.0000 - val_tn0.3: 15917554.0000 - val_fn0.3: 253552.0000 - val_precision0.3: 0.8337 - val_recall0.3: 0.9198 - val_tp0.5: 2799018
    .0000 - val_fp0.5: 344609.0000 - val_tn0.5: 16153337.0000 - val_fn0.5: 363836.0000 - val_precision0.5: 0.8904 - val_recall0.5: 0.8850 - val_tp0.7: 2633849.0000 - val_fp0.7: 176599.0000 - v
    al_tn0.7: 16321347.0000 - val_fn0.7: 529005.0000 - val_precision0.7: 0.9372 - val_recall0.7: 0.8327 - val_tp0.9: 2302923.0000 - val_fp0.9: 54090.0000 - val_tn0.9: 16443856.0000 - val_fn0.9
    : 859931.0000 - val_precision0.9: 0.9771 - val_recall0.9: 0.7281 - val_accuracy: 0.9640 - val_auc: 0.9722 - val_f1: 0.2772
    Epoch 19/20
    240/240 [==============================] - 123s 514ms/step - loss: 0.2516 - tp0.1: 11404546.0000 - fp0.1: 4602276.0000 - tn0.1: 62071184.0000 - fn0.1: 565221.0000 - precision0.1: 0.7125 -
    recall0.1: 0.9528 - tp0.3: 10964712.0000 - fp0.3: 2131443.0000 - tn0.3: 64541984.0000 - fn0.3: 1005055.0000 - precision0.3: 0.8372 - recall0.3: 0.9160 - tp0.5: 10530069.0000 - fp0.5: 11977
    48.0000 - tn0.5: 65475676.0000 - fn0.5: 1439698.0000 - precision0.5: 0.8979 - recall0.5: 0.8797 - tp0.7: 9887845.0000 - fp0.7: 573172.0000 - tn0.7: 66100272.0000 - fn0.7: 2081922.0000 - pr
    ecision0.7: 0.9452 - recall0.7: 0.8261 - tp0.9: 8654050.0000 - fp0.9: 150095.0000 - tn0.9: 66523328.0000 - fn0.9: 3315717.0000 - precision0.9: 0.9830 - recall0.9: 0.7230 - accuracy: 0.9665
     - auc: 0.9717 - f1: 0.2642 - val_loss: 0.2481 - val_tp0.1: 3008011.0000 - val_fp0.1: 953689.0000 - val_tn0.1: 15544257.0000 - val_fn0.1: 154843.0000 - val_precision0.1: 0.7593 - val_recal
    l0.1: 0.9510 - val_tp0.3: 2881653.0000 - val_fp0.3: 443165.0000 - val_tn0.3: 16054781.0000 - val_fn0.3: 281201.0000 - val_precision0.3: 0.8667 - val_recall0.3: 0.9111 - val_tp0.5: 2768505.
    0000 - val_fp0.5: 264471.0000 - val_tn0.5: 16233475.0000 - val_fn0.5: 394349.0000 - val_precision0.5: 0.9128 - val_recall0.5: 0.8753 - val_tp0.7: 2606170.0000 - val_fp0.7: 136595.0000 - va
    l_tn0.7: 16361351.0000 - val_fn0.7: 556684.0000 - val_precision0.7: 0.9502 - val_recall0.7: 0.8240 - val_tp0.9: 2281582.0000 - val_fp0.9: 42534.0000 - val_tn0.9: 16455412.0000 - val_fn0.9:
     881272.0000 - val_precision0.9: 0.9817 - val_recall0.9: 0.7214 - val_accuracy: 0.9665 - val_auc: 0.9714 - val_f1: 0.2772
    Epoch 20/20
    240/240 [==============================] - 123s 513ms/step - loss: 0.2463 - tp0.1: 11418792.0000 - fp0.1: 4532019.0000 - tn0.1: 62141424.0000 - fn0.1: 550975.0000 - precision0.1: 0.7159 -
    recall0.1: 0.9540 - tp0.3: 10983398.0000 - fp0.3: 2088662.0000 - tn0.3: 64584768.0000 - fn0.3: 986369.0000 - precision0.3: 0.8402 - recall0.3: 0.9176 - tp0.5: 10555722.0000 - fp0.5: 118115
    9.0000 - tn0.5: 65492296.0000 - fn0.5: 1414045.0000 - precision0.5: 0.8994 - recall0.5: 0.8819 - tp0.7: 9935653.0000 - fp0.7: 569774.0000 - tn0.7: 66103648.0000 - fn0.7: 2034114.0000 - pre
    cision0.7: 0.9458 - recall0.7: 0.8301 - tp0.9: 8727770.0000 - fp0.9: 149122.0000 - tn0.9: 66524300.0000 - fn0.9: 3241997.0000 - precision0.9: 0.9832 - recall0.9: 0.7292 - accuracy: 0.9670
    - auc: 0.9724 - f1: 0.2642 - val_loss: 0.2428 - val_tp0.1: 3043089.0000 - val_fp0.1: 1254723.0000 - val_tn0.1: 15243223.0000 - val_fn0.1: 119765.0000 - val_precision0.1: 0.7081 - val_recal
    l0.1: 0.9621 - val_tp0.3: 2938943.0000 - val_fp0.3: 582622.0000 - val_tn0.3: 15915324.0000 - val_fn0.3: 223911.0000 - val_precision0.3: 0.8346 - val_recall0.3: 0.9292 - val_tp0.5: 2838207.
    0000 - val_fp0.5: 352144.0000 - val_tn0.5: 16145802.0000 - val_fn0.5: 324647.0000 - val_precision0.5: 0.8896 - val_recall0.5: 0.8974 - val_tp0.7: 2688032.0000 - val_fp0.7: 185761.0000 - va
    l_tn0.7: 16312185.0000 - val_fn0.7: 474822.0000 - val_precision0.7: 0.9354 - val_recall0.7: 0.8499 - val_tp0.9: 2371174.0000 - val_fp0.9: 57559.0000 - val_tn0.9: 16440387.0000 - val_fn0.9:
     791680.0000 - val_precision0.9: 0.9763 - val_recall0.9: 0.7497 - val_accuracy: 0.9656 - val_auc: 0.9761 - val_f1: 0.2772
    --- Running training session 62/140
    {'hp_epochs': 20, 'hp_batch_size': 20, 'hp_scaler': 'standard', 'hp_n_levels': 3, 'hp_first_filters': 128, 'hp_pool_size': 4, 'hp_input_size': 16384, 'hp_lr_start': 0.04354970735327304, 'h
    p_lr_power': 1.0}
    --- repeat #: 2
    input - shape:   (None, 16384, 1)
    output - shape:  (None, 16384, 1)
    Epoch 1/20
    240/240 [==============================] - 144s 534ms/step - loss: 1.1479 - tp0.1: 9906728.0000 - fp0.1: 13187183.0000 - tn0.1: 53486260.0000 - fn0.1: 2063039.0000 - precision0.1: 0.4290 -
     recall0.1: 0.8276 - tp0.3: 8671992.0000 - fp0.3: 5127913.0000 - tn0.3: 61545512.0000 - fn0.3: 3297775.0000 - precision0.3: 0.6284 - recall0.3: 0.7245 - tp0.5: 7568431.0000 - fp0.5: 253412
    2.0000 - tn0.5: 64139300.0000 - fn0.5: 4401336.0000 - precision0.5: 0.7492 - recall0.5: 0.6323 - tp0.7: 6054670.0000 - fp0.7: 1016029.0000 - tn0.7: 65657384.0000 - fn0.7: 5915097.0000 - pr
    ecision0.7: 0.8563 - recall0.7: 0.5058 - tp0.9: 3696361.0000 - fp0.9: 210116.0000 - tn0.9: 66463280.0000 - fn0.9: 8273406.0000 - precision0.9: 0.9462 - recall0.9: 0.3088 - accuracy: 0.9118
     - auc: 0.8839 - f1: 0.2642 - val_loss: 1.3309 - val_tp0.1: 1733257.0000 - val_fp0.1: 2884942.0000 - val_tn0.1: 13613004.0000 - val_fn0.1: 1429597.0000 - val_precision0.1: 0.3753 - val_rec
    all0.1: 0.5480 - val_tp0.3: 1371992.0000 - val_fp0.3: 689902.0000 - val_tn0.3: 15808044.0000 - val_fn0.3: 1790862.0000 - val_precision0.3: 0.6654 - val_recall0.3: 0.4338 - val_tp0.5: 12399
    93.0000 - val_fp0.5: 314549.0000 - val_tn0.5: 16183397.0000 - val_fn0.5: 1922861.0000 - val_precision0.5: 0.7977 - val_recall0.5: 0.3920 - val_tp0.7: 1088844.0000 - val_fp0.7: 172991.0000
    - val_tn0.7: 16324955.0000 - val_fn0.7: 2074010.0000 - val_precision0.7: 0.8629 - val_recall0.7: 0.3443 - val_tp0.9: 854487.0000 - val_fp0.9: 85499.0000 - val_tn0.9: 16412447.0000 - val_fn
    0.9: 2308367.0000 - val_precision0.9: 0.9090 - val_recall0.9: 0.2702 - val_accuracy: 0.8862 - val_auc: 0.7423 - val_f1: 0.2772
    Epoch 2/20
    240/240 [==============================] - 124s 516ms/step - loss: 0.8128 - tp0.1: 10386764.0000 - fp0.1: 10662276.0000 - tn0.1: 56011152.0000 - fn0.1: 1583003.0000 - precision0.1: 0.4935
    - recall0.1: 0.8677 - tp0.3: 9374552.0000 - fp0.3: 4301798.0000 - tn0.3: 62371648.0000 - fn0.3: 2595215.0000 - precision0.3: 0.6855 - recall0.3: 0.7832 - tp0.5: 8455753.0000 - fp0.5: 22081
    36.0000 - tn0.5: 64465300.0000 - fn0.5: 3514014.0000 - precision0.5: 0.7929 - recall0.5: 0.7064 - tp0.7: 7249754.0000 - fp0.7: 1037037.0000 - tn0.7: 65636376.0000 - fn0.7: 4720013.0000 - p
    recision0.7: 0.8749 - recall0.7: 0.6057 - tp0.9: 4971364.0000 - fp0.9: 253682.0000 - tn0.9: 66419772.0000 - fn0.9: 6998403.0000 - precision0.9: 0.9514 - recall0.9: 0.4153 - accuracy: 0.927
    2 - auc: 0.9156 - f1: 0.2642 - val_loss: 1.7318 - val_tp0.1: 1429651.0000 - val_fp0.1: 93149.0000 - val_tn0.1: 16404797.0000 - val_fn0.1: 1733203.0000 - val_precision0.1: 0.9388 - val_reca
    ll0.1: 0.4520 - val_tp0.3: 1260333.0000 - val_fp0.3: 61973.0000 - val_tn0.3: 16435973.0000 - val_fn0.3: 1902521.0000 - val_precision0.3: 0.9531 - val_recall0.3: 0.3985 - val_tp0.5: 1164560
    .0000 - val_fp0.5: 49155.0000 - val_tn0.5: 16448791.0000 - val_fn0.5: 1998294.0000 - val_precision0.5: 0.9595 - val_recall0.5: 0.3682 - val_tp0.7: 1052319.0000 - val_fp0.7: 36199.0000 - va
    l_tn0.7: 16461747.0000 - val_fn0.7: 2110535.0000 - val_precision0.7: 0.9667 - val_recall0.7: 0.3327 - val_tp0.9: 812871.0000 - val_fp0.9: 16636.0000 - val_tn0.9: 16481310.0000 - val_fn0.9:
     2349983.0000 - val_precision0.9: 0.9799 - val_recall0.9: 0.2570 - val_accuracy: 0.8959 - val_auc: 0.7346 - val_f1: 0.2772
    Epoch 3/20
    240/240 [==============================] - 124s 516ms/step - loss: 0.5249 - tp0.1: 10650727.0000 - fp0.1: 8971663.0000 - tn0.1: 57701744.0000 - fn0.1: 1319040.0000 - precision0.1: 0.5428 -
     recall0.1: 0.8898 - tp0.3: 9707938.0000 - fp0.3: 3584621.0000 - tn0.3: 63088792.0000 - fn0.3: 2261829.0000 - precision0.3: 0.7303 - recall0.3: 0.8110 - tp0.5: 8947919.0000 - fp0.5: 196644
    3.0000 - tn0.5: 64706976.0000 - fn0.5: 3021848.0000 - precision0.5: 0.8198 - recall0.5: 0.7475 - tp0.7: 7846298.0000 - fp0.7: 941117.0000 - tn0.7: 65732308.0000 - fn0.7: 4123469.0000 - pre
    cision0.7: 0.8929 - recall0.7: 0.6555 - tp0.9: 5813133.0000 - fp0.9: 247129.0000 - tn0.9: 66426308.0000 - fn0.9: 6156634.0000 - precision0.9: 0.9592 - recall0.9: 0.4857 - accuracy: 0.9366
    - auc: 0.9272 - f1: 0.2642 - val_loss: 0.6610 - val_tp0.1: 2455351.0000 - val_fp0.1: 660521.0000 - val_tn0.1: 15837425.0000 - val_fn0.1: 707503.0000 - val_precision0.1: 0.7880 - val_recall
    0.1: 0.7763 - val_tp0.3: 2077241.0000 - val_fp0.3: 245933.0000 - val_tn0.3: 16252013.0000 - val_fn0.3: 1085613.0000 - val_precision0.3: 0.8941 - val_recall0.3: 0.6568 - val_tp0.5: 1903683.
    0000 - val_fp0.5: 169535.0000 - val_tn0.5: 16328411.0000 - val_fn0.5: 1259171.0000 - val_precision0.5: 0.9182 - val_recall0.5: 0.6019 - val_tp0.7: 1686781.0000 - val_fp0.7: 108097.0000 - v
    al_tn0.7: 16389849.0000 - val_fn0.7: 1476073.0000 - val_precision0.7: 0.9398 - val_recall0.7: 0.5333 - val_tp0.9: 1239931.0000 - val_fp0.9: 39608.0000 - val_tn0.9: 16458338.0000 - val_fn0.
    9: 1922923.0000 - val_precision0.9: 0.9690 - val_recall0.9: 0.3920 - val_accuracy: 0.9273 - val_auc: 0.8923 - val_f1: 0.2772
    Epoch 4/20
    240/240 [==============================] - 124s 517ms/step - loss: 0.4701 - tp0.1: 10831550.0000 - fp0.1: 8197166.0000 - tn0.1: 58476288.0000 - fn0.1: 1138217.0000 - precision0.1: 0.5692 -
     recall0.1: 0.9049 - tp0.3: 10003164.0000 - fp0.3: 3493324.0000 - tn0.3: 63180092.0000 - fn0.3: 1966603.0000 - precision0.3: 0.7412 - recall0.3: 0.8357 - tp0.5: 9170982.0000 - fp0.5: 17747
    67.0000 - tn0.5: 64898668.0000 - fn0.5: 2798785.0000 - precision0.5: 0.8379 - recall0.5: 0.7662 - tp0.7: 8179901.0000 - fp0.7: 877946.0000 - tn0.7: 65795468.0000 - fn0.7: 3789866.0000 - pr
    ecision0.7: 0.9031 - recall0.7: 0.6834 - tp0.9: 6375174.0000 - fp0.9: 261023.0000 - tn0.9: 66412392.0000 - fn0.9: 5594593.0000 - precision0.9: 0.9607 - recall0.9: 0.5326 - accuracy: 0.9418
     - auc: 0.9378 - f1: 0.2642 - val_loss: 1.3899 - val_tp0.1: 3117692.0000 - val_fp0.1: 9912153.0000 - val_tn0.1: 6585793.0000 - val_fn0.1: 45162.0000 - val_precision0.1: 0.2393 - val_recall
    0.1: 0.9857 - val_tp0.3: 3079025.0000 - val_fp0.3: 7898193.0000 - val_tn0.3: 8599753.0000 - val_fn0.3: 83829.0000 - val_precision0.3: 0.2805 - val_recall0.3: 0.9735 - val_tp0.5: 3048909.00
    00 - val_fp0.5: 6769491.0000 - val_tn0.5: 9728455.0000 - val_fn0.5: 113945.0000 - val_precision0.5: 0.3105 - val_recall0.5: 0.9640 - val_tp0.7: 2994971.0000 - val_fp0.7: 5019595.0000 - val
    _tn0.7: 11478351.0000 - val_fn0.7: 167883.0000 - val_precision0.7: 0.3737 - val_recall0.7: 0.9469 - val_tp0.9: 2874370.0000 - val_fp0.9: 2531864.0000 - val_tn0.9: 13966082.0000 - val_fn0.9
    : 288484.0000 - val_precision0.9: 0.5317 - val_recall0.9: 0.9088 - val_accuracy: 0.6499 - val_auc: 0.9447 - val_f1: 0.2772
    Epoch 5/20
    240/240 [==============================] - 124s 517ms/step - loss: 0.4466 - tp0.1: 10915404.0000 - fp0.1: 7994592.0000 - tn0.1: 58678848.0000 - fn0.1: 1054363.0000 - precision0.1: 0.5772 -
     recall0.1: 0.9119 - tp0.3: 10101214.0000 - fp0.3: 3354053.0000 - tn0.3: 63319376.0000 - fn0.3: 1868553.0000 - precision0.3: 0.7507 - recall0.3: 0.8439 - tp0.5: 9363723.0000 - fp0.5: 17904
    83.0000 - tn0.5: 64882964.0000 - fn0.5: 2606044.0000 - precision0.5: 0.8395 - recall0.5: 0.7823 - tp0.7: 8326836.0000 - fp0.7: 857418.0000 - tn0.7: 65816000.0000 - fn0.7: 3642931.0000 - pr
    ecision0.7: 0.9066 - recall0.7: 0.6957 - tp0.9: 6486870.0000 - fp0.9: 255970.0000 - tn0.9: 66417480.0000 - fn0.9: 5482897.0000 - precision0.9: 0.9620 - recall0.9: 0.5419 - accuracy: 0.9441
     - auc: 0.9425 - f1: 0.2642 - val_loss: 0.9230 - val_tp0.1: 3093810.0000 - val_fp0.1: 6728709.0000 - val_tn0.1: 9769237.0000 - val_fn0.1: 69044.0000 - val_precision0.1: 0.3150 - val_recall
    0.1: 0.9782 - val_tp0.3: 2983445.0000 - val_fp0.3: 3338469.0000 - val_tn0.3: 13159477.0000 - val_fn0.3: 179409.0000 - val_precision0.3: 0.4719 - val_recall0.3: 0.9433 - val_tp0.5: 2913456.
    0000 - val_fp0.5: 2448590.0000 - val_tn0.5: 14049356.0000 - val_fn0.5: 249398.0000 - val_precision0.5: 0.5433 - val_recall0.5: 0.9211 - val_tp0.7: 2826495.0000 - val_fp0.7: 1728169.0000 -
    val_tn0.7: 14769777.0000 - val_fn0.7: 336359.0000 - val_precision0.7: 0.6206 - val_recall0.7: 0.8937 - val_tp0.9: 2684311.0000 - val_fp0.9: 1097331.0000 - val_tn0.9: 15400615.0000 - val_fn
    0.9: 478543.0000 - val_precision0.9: 0.7098 - val_recall0.9: 0.8487 - val_accuracy: 0.8628 - val_auc: 0.9474 - val_f1: 0.2772
    Epoch 6/20
    240/240 [==============================] - 124s 516ms/step - loss: 0.4071 - tp0.1: 11022248.0000 - fp0.1: 7374717.0000 - tn0.1: 59298728.0000 - fn0.1: 947519.0000 - precision0.1: 0.5991 -
    recall0.1: 0.9208 - tp0.3: 10241896.0000 - fp0.3: 3036735.0000 - tn0.3: 63636704.0000 - fn0.3: 1727871.0000 - precision0.3: 0.7713 - recall0.3: 0.8556 - tp0.5: 9584629.0000 - fp0.5: 165806
    9.0000 - tn0.5: 65015368.0000 - fn0.5: 2385138.0000 - precision0.5: 0.8525 - recall0.5: 0.8007 - tp0.7: 8694782.0000 - fp0.7: 820555.0000 - tn0.7: 65852888.0000 - fn0.7: 3274985.0000 - pre
    cision0.7: 0.9138 - recall0.7: 0.7264 - tp0.9: 6966857.0000 - fp0.9: 241359.0000 - tn0.9: 66432068.0000 - fn0.9: 5002910.0000 - precision0.9: 0.9665 - recall0.9: 0.5820 - accuracy: 0.9486
    - auc: 0.9486 - f1: 0.2642 - val_loss: 0.4390 - val_tp0.1: 2785859.0000 - val_fp0.1: 688884.0000 - val_tn0.1: 15809062.0000 - val_fn0.1: 376995.0000 - val_precision0.1: 0.8017 - val_recall
    0.1: 0.8808 - val_tp0.3: 2458538.0000 - val_fp0.3: 247679.0000 - val_tn0.3: 16250267.0000 - val_fn0.3: 704316.0000 - val_precision0.3: 0.9085 - val_recall0.3: 0.7773 - val_tp0.5: 2155525.0
    000 - val_fp0.5: 109573.0000 - val_tn0.5: 16388373.0000 - val_fn0.5: 1007329.0000 - val_precision0.5: 0.9516 - val_recall0.5: 0.6815 - val_tp0.7: 1802828.0000 - val_fp0.7: 45112.0000 - val
    _tn0.7: 16452834.0000 - val_fn0.7: 1360026.0000 - val_precision0.7: 0.9756 - val_recall0.7: 0.5700 - val_tp0.9: 1314546.0000 - val_fp0.9: 12049.0000 - val_tn0.9: 16485897.0000 - val_fn0.9:
     1848308.0000 - val_precision0.9: 0.9909 - val_recall0.9: 0.4156 - val_accuracy: 0.9432 - val_auc: 0.9435 - val_f1: 0.2772
    Epoch 7/20
    240/240 [==============================] - 124s 516ms/step - loss: 0.3853 - tp0.1: 11079386.0000 - fp0.1: 6940988.0000 - tn0.1: 59732460.0000 - fn0.1: 890381.0000 - precision0.1: 0.6148 -
    recall0.1: 0.9256 - tp0.3: 10346458.0000 - fp0.3: 3000615.0000 - tn0.3: 63672804.0000 - fn0.3: 1623309.0000 - precision0.3: 0.7752 - recall0.3: 0.8644 - tp0.5: 9690133.0000 - fp0.5: 161636
    9.0000 - tn0.5: 65057072.0000 - fn0.5: 2279634.0000 - precision0.5: 0.8570 - recall0.5: 0.8096 - tp0.7: 8843112.0000 - fp0.7: 822086.0000 - tn0.7: 65851372.0000 - fn0.7: 3126655.0000 - pre
    cision0.7: 0.9149 - recall0.7: 0.7388 - tp0.9: 7135245.0000 - fp0.9: 238695.0000 - tn0.9: 66434744.0000 - fn0.9: 4834522.0000 - precision0.9: 0.9676 - recall0.9: 0.5961 - accuracy: 0.9505
    - auc: 0.9518 - f1: 0.2642 - val_loss: 0.4072 - val_tp0.1: 3071232.0000 - val_fp0.1: 3190431.0000 - val_tn0.1: 13307515.0000 - val_fn0.1: 91622.0000 - val_precision0.1: 0.4905 - val_recall
    0.1: 0.9710 - val_tp0.3: 2969302.0000 - val_fp0.3: 1516252.0000 - val_tn0.3: 14981694.0000 - val_fn0.3: 193552.0000 - val_precision0.3: 0.6620 - val_recall0.3: 0.9388 - val_tp0.5: 2813331.
    0000 - val_fp0.5: 766580.0000 - val_tn0.5: 15731366.0000 - val_fn0.5: 349523.0000 - val_precision0.5: 0.7859 - val_recall0.5: 0.8895 - val_tp0.7: 2565842.0000 - val_fp0.7: 340498.0000 - va
    l_tn0.7: 16157448.0000 - val_fn0.7: 597012.0000 - val_precision0.7: 0.8828 - val_recall0.7: 0.8112 - val_tp0.9: 2014990.0000 - val_fp0.9: 79766.0000 - val_tn0.9: 16418180.0000 - val_fn0.9:
     1147864.0000 - val_precision0.9: 0.9619 - val_recall0.9: 0.6371 - val_accuracy: 0.9432 - val_auc: 0.9709 - val_f1: 0.2772
    Epoch 8/20
    240/240 [==============================] - 124s 517ms/step - loss: 0.3633 - tp0.1: 11135897.0000 - fp0.1: 6660915.0000 - tn0.1: 60012532.0000 - fn0.1: 833870.0000 - precision0.1: 0.6257 -
    recall0.1: 0.9303 - tp0.3: 10437033.0000 - fp0.3: 2736926.0000 - tn0.3: 63936496.0000 - fn0.3: 1532734.0000 - precision0.3: 0.7922 - recall0.3: 0.8719 - tp0.5: 9858968.0000 - fp0.5: 152258
    9.0000 - tn0.5: 65150856.0000 - fn0.5: 2110799.0000 - precision0.5: 0.8662 - recall0.5: 0.8237 - tp0.7: 9048624.0000 - fp0.7: 753519.0000 - tn0.7: 65919908.0000 - fn0.7: 2921143.0000 - pre
    cision0.7: 0.9231 - recall0.7: 0.7560 - tp0.9: 7412042.0000 - fp0.9: 213551.0000 - tn0.9: 66459872.0000 - fn0.9: 4557725.0000 - precision0.9: 0.9720 - recall0.9: 0.6192 - accuracy: 0.9538
    - auc: 0.9558 - f1: 0.2642 - val_loss: 0.4570 - val_tp0.1: 3014123.0000 - val_fp0.1: 2699685.0000 - val_tn0.1: 13798261.0000 - val_fn0.1: 148731.0000 - val_precision0.1: 0.5275 - val_recal
    l0.1: 0.9530 - val_tp0.3: 2936029.0000 - val_fp0.3: 1673840.0000 - val_tn0.3: 14824106.0000 - val_fn0.3: 226825.0000 - val_precision0.3: 0.6369 - val_recall0.3: 0.9283 - val_tp0.5: 2861446
    .0000 - val_fp0.5: 1230626.0000 - val_tn0.5: 15267320.0000 - val_fn0.5: 301408.0000 - val_precision0.5: 0.6993 - val_recall0.5: 0.9047 - val_tp0.7: 2733900.0000 - val_fp0.7: 781063.0000 -
    val_tn0.7: 15716883.0000 - val_fn0.7: 428954.0000 - val_precision0.7: 0.7778 - val_recall0.7: 0.8644 - val_tp0.9: 2482216.0000 - val_fp0.9: 390029.0000 - val_tn0.9: 16107917.0000 - val_fn0
    .9: 680638.0000 - val_precision0.9: 0.8642 - val_recall0.9: 0.7848 - val_accuracy: 0.9221 - val_auc: 0.9600 - val_f1: 0.2772
    Epoch 9/20
    240/240 [==============================] - 124s 518ms/step - loss: 0.3559 - tp0.1: 11134313.0000 - fp0.1: 6236820.0000 - tn0.1: 60436612.0000 - fn0.1: 835454.0000 - precision0.1: 0.6410 -
    recall0.1: 0.9302 - tp0.3: 10469952.0000 - fp0.3: 2640391.0000 - tn0.3: 64033040.0000 - fn0.3: 1499815.0000 - precision0.3: 0.7986 - recall0.3: 0.8747 - tp0.5: 9934376.0000 - fp0.5: 152988
    9.0000 - tn0.5: 65143560.0000 - fn0.5: 2035391.0000 - precision0.5: 0.8666 - recall0.5: 0.8300 - tp0.7: 9136907.0000 - fp0.7: 753965.0000 - tn0.7: 65919460.0000 - fn0.7: 2832860.0000 - pre
    cision0.7: 0.9238 - recall0.7: 0.7633 - tp0.9: 7562687.0000 - fp0.9: 213280.0000 - tn0.9: 66460156.0000 - fn0.9: 4407080.0000 - precision0.9: 0.9726 - recall0.9: 0.6318 - accuracy: 0.9547
    - auc: 0.9561 - f1: 0.2642 - val_loss: 0.3791 - val_tp0.1: 2856464.0000 - val_fp0.1: 956306.0000 - val_tn0.1: 15541640.0000 - val_fn0.1: 306390.0000 - val_precision0.1: 0.7492 - val_recall
    0.1: 0.9031 - val_tp0.3: 2644444.0000 - val_fp0.3: 392043.0000 - val_tn0.3: 16105903.0000 - val_fn0.3: 518410.0000 - val_precision0.3: 0.8709 - val_recall0.3: 0.8361 - val_tp0.5: 2503008.0
    000 - val_fp0.5: 246935.0000 - val_tn0.5: 16251011.0000 - val_fn0.5: 659846.0000 - val_precision0.5: 0.9102 - val_recall0.5: 0.7914 - val_tp0.7: 2293518.0000 - val_fp0.7: 135876.0000 - val
    _tn0.7: 16362070.0000 - val_fn0.7: 869336.0000 - val_precision0.7: 0.9441 - val_recall0.7: 0.7251 - val_tp0.9: 1878107.0000 - val_fp0.9: 37476.0000 - val_tn0.9: 16460470.0000 - val_fn0.9:
    1284747.0000 - val_precision0.9: 0.9804 - val_recall0.9: 0.5938 - val_accuracy: 0.9539 - val_auc: 0.9458 - val_f1: 0.2772
    Epoch 10/20
    240/240 [==============================] - 124s 517ms/step - loss: 0.3399 - tp0.1: 11192783.0000 - fp0.1: 6148473.0000 - tn0.1: 60524940.0000 - fn0.1: 776984.0000 - precision0.1: 0.6454 -
    recall0.1: 0.9351 - tp0.3: 10562088.0000 - fp0.3: 2647342.0000 - tn0.3: 64026076.0000 - fn0.3: 1407679.0000 - precision0.3: 0.7996 - recall0.3: 0.8824 - tp0.5: 10019446.0000 - fp0.5: 14919
    46.0000 - tn0.5: 65181488.0000 - fn0.5: 1950321.0000 - precision0.5: 0.8704 - recall0.5: 0.8371 - tp0.7: 9263641.0000 - fp0.7: 747160.0000 - tn0.7: 65926272.0000 - fn0.7: 2706126.0000 - pr
    ecision0.7: 0.9254 - recall0.7: 0.7739 - tp0.9: 7720186.0000 - fp0.9: 209487.0000 - tn0.9: 66463960.0000 - fn0.9: 4249581.0000 - precision0.9: 0.9736 - recall0.9: 0.6450 - accuracy: 0.9562
     - auc: 0.9595 - f1: 0.2642 - val_loss: 0.3522 - val_tp0.1: 3064550.0000 - val_fp0.1: 2487874.0000 - val_tn0.1: 14010072.0000 - val_fn0.1: 98304.0000 - val_precision0.1: 0.5519 - val_recal
    l0.1: 0.9689 - val_tp0.3: 2948761.0000 - val_fp0.3: 1064135.0000 - val_tn0.3: 15433811.0000 - val_fn0.3: 214093.0000 - val_precision0.3: 0.7348 - val_recall0.3: 0.9323 - val_tp0.5: 2865468
    .0000 - val_fp0.5: 718289.0000 - val_tn0.5: 15779657.0000 - val_fn0.5: 297386.0000 - val_precision0.5: 0.7996 - val_recall0.5: 0.9060 - val_tp0.7: 2741942.0000 - val_fp0.7: 426555.0000 - v
    al_tn0.7: 16071391.0000 - val_fn0.7: 420912.0000 - val_precision0.7: 0.8654 - val_recall0.7: 0.8669 - val_tp0.9: 2476605.0000 - val_fp0.9: 171677.0000 - val_tn0.9: 16326269.0000 - val_fn0.
    9: 686249.0000 - val_precision0.9: 0.9352 - val_recall0.9: 0.7830 - val_accuracy: 0.9483 - val_auc: 0.9739 - val_f1: 0.2772
    Epoch 11/20
    240/240 [==============================] - 124s 517ms/step - loss: 0.3283 - tp0.1: 11237515.0000 - fp0.1: 6025317.0000 - tn0.1: 60648124.0000 - fn0.1: 732252.0000 - precision0.1: 0.6510 -
    recall0.1: 0.9388 - tp0.3: 10617524.0000 - fp0.3: 2587448.0000 - tn0.3: 64085984.0000 - fn0.3: 1352243.0000 - precision0.3: 0.8041 - recall0.3: 0.8870 - tp0.5: 10090900.0000 - fp0.5: 14806
    86.0000 - tn0.5: 65192740.0000 - fn0.5: 1878867.0000 - precision0.5: 0.8720 - recall0.5: 0.8430 - tp0.7: 9347944.0000 - fp0.7: 746924.0000 - tn0.7: 65926508.0000 - fn0.7: 2621823.0000 - pr
    ecision0.7: 0.9260 - recall0.7: 0.7810 - tp0.9: 7842670.0000 - fp0.9: 222008.0000 - tn0.9: 66451408.0000 - fn0.9: 4127097.0000 - precision0.9: 0.9725 - recall0.9: 0.6552 - accuracy: 0.9573
     - auc: 0.9614 - f1: 0.2642 - val_loss: 0.3862 - val_tp0.1: 3085105.0000 - val_fp0.1: 3480345.0000 - val_tn0.1: 13017601.0000 - val_fn0.1: 77749.0000 - val_precision0.1: 0.4699 - val_recal
    l0.1: 0.9754 - val_tp0.3: 2931914.0000 - val_fp0.3: 986226.0000 - val_tn0.3: 15511720.0000 - val_fn0.3: 230940.0000 - val_precision0.3: 0.7483 - val_recall0.3: 0.9270 - val_tp0.5: 2759869.
    0000 - val_fp0.5: 439560.0000 - val_tn0.5: 16058386.0000 - val_fn0.5: 402985.0000 - val_precision0.5: 0.8626 - val_recall0.5: 0.8726 - val_tp0.7: 2487808.0000 - val_fp0.7: 172315.0000 - va
    l_tn0.7: 16325631.0000 - val_fn0.7: 675046.0000 - val_precision0.7: 0.9352 - val_recall0.7: 0.7866 - val_tp0.9: 1923553.0000 - val_fp0.9: 34623.0000 - val_tn0.9: 16463323.0000 - val_fn0.9:
     1239301.0000 - val_precision0.9: 0.9823 - val_recall0.9: 0.6082 - val_accuracy: 0.9571 - val_auc: 0.9755 - val_f1: 0.2772
    Epoch 12/20
    240/240 [==============================] - 124s 517ms/step - loss: 0.3149 - tp0.1: 11244552.0000 - fp0.1: 5636478.0000 - tn0.1: 61036940.0000 - fn0.1: 725215.0000 - precision0.1: 0.6661 -
    recall0.1: 0.9394 - tp0.3: 10645433.0000 - fp0.3: 2370845.0000 - tn0.3: 64302560.0000 - fn0.3: 1324334.0000 - precision0.3: 0.8179 - recall0.3: 0.8894 - tp0.5: 10139252.0000 - fp0.5: 13412
    79.0000 - tn0.5: 65332156.0000 - fn0.5: 1830515.0000 - precision0.5: 0.8832 - recall0.5: 0.8471 - tp0.7: 9434975.0000 - fp0.7: 668796.0000 - tn0.7: 66004652.0000 - fn0.7: 2534792.0000 - pr
    ecision0.7: 0.9338 - recall0.7: 0.7882 - tp0.9: 8010285.0000 - fp0.9: 191192.0000 - tn0.9: 66482252.0000 - fn0.9: 3959482.0000 - precision0.9: 0.9767 - recall0.9: 0.6692 - accuracy: 0.9597
     - auc: 0.9629 - f1: 0.2642 - val_loss: 0.3954 - val_tp0.1: 3008877.0000 - val_fp0.1: 2662719.0000 - val_tn0.1: 13835227.0000 - val_fn0.1: 153977.0000 - val_precision0.1: 0.5305 - val_reca
    ll0.1: 0.9513 - val_tp0.3: 2890670.0000 - val_fp0.3: 1167312.0000 - val_tn0.3: 15330634.0000 - val_fn0.3: 272184.0000 - val_precision0.3: 0.7123 - val_recall0.3: 0.9139 - val_tp0.5: 279631
    0.0000 - val_fp0.5: 773358.0000 - val_tn0.5: 15724588.0000 - val_fn0.5: 366544.0000 - val_precision0.5: 0.7834 - val_recall0.5: 0.8841 - val_tp0.7: 2635367.0000 - val_fp0.7: 412051.0000 -
    val_tn0.7: 16085895.0000 - val_fn0.7: 527487.0000 - val_precision0.7: 0.8648 - val_recall0.7: 0.8332 - val_tp0.9: 2302238.0000 - val_fp0.9: 140209.0000 - val_tn0.9: 16357737.0000 - val_fn0
    .9: 860616.0000 - val_precision0.9: 0.9426 - val_recall0.9: 0.7279 - val_accuracy: 0.9420 - val_auc: 0.9629 - val_f1: 0.2772
    Epoch 13/20
    240/240 [==============================] - 124s 516ms/step - loss: 0.3001 - tp0.1: 11285735.0000 - fp0.1: 5401576.0000 - tn0.1: 61271836.0000 - fn0.1: 684032.0000 - precision0.1: 0.6763 -
    recall0.1: 0.9429 - tp0.3: 10728442.0000 - fp0.3: 2362125.0000 - tn0.3: 64311320.0000 - fn0.3: 1241325.0000 - precision0.3: 0.8196 - recall0.3: 0.8963 - tp0.5: 10253499.0000 - fp0.5: 13693
    34.0000 - tn0.5: 65304092.0000 - fn0.5: 1716268.0000 - precision0.5: 0.8822 - recall0.5: 0.8566 - tp0.7: 9558014.0000 - fp0.7: 688333.0000 - tn0.7: 65985076.0000 - fn0.7: 2411753.0000 - pr
    ecision0.7: 0.9328 - recall0.7: 0.7985 - tp0.9: 8134971.0000 - fp0.9: 193269.0000 - tn0.9: 66480164.0000 - fn0.9: 3834796.0000 - precision0.9: 0.9768 - recall0.9: 0.6796 - accuracy: 0.9608
     - auc: 0.9650 - f1: 0.2642 - val_loss: 0.3591 - val_tp0.1: 2894724.0000 - val_fp0.1: 841367.0000 - val_tn0.1: 15656579.0000 - val_fn0.1: 268130.0000 - val_precision0.1: 0.7748 - val_recal
    l0.1: 0.9152 - val_tp0.3: 2569647.0000 - val_fp0.3: 243687.0000 - val_tn0.3: 16254259.0000 - val_fn0.3: 593207.0000 - val_precision0.3: 0.9134 - val_recall0.3: 0.8124 - val_tp0.5: 2323874.
    0000 - val_fp0.5: 112189.0000 - val_tn0.5: 16385757.0000 - val_fn0.5: 838980.0000 - val_precision0.5: 0.9539 - val_recall0.5: 0.7347 - val_tp0.7: 1996399.0000 - val_fp0.7: 37554.0000 - val
    _tn0.7: 16460392.0000 - val_fn0.7: 1166455.0000 - val_precision0.7: 0.9815 - val_recall0.7: 0.6312 - val_tp0.9: 1499767.0000 - val_fp0.9: 5568.0000 - val_tn0.9: 16492378.0000 - val_fn0.9:
    1663087.0000 - val_precision0.9: 0.9963 - val_recall0.9: 0.4742 - val_accuracy: 0.9516 - val_auc: 0.9540 - val_f1: 0.2772
    Epoch 14/20
    240/240 [==============================] - 124s 516ms/step - loss: 0.2927 - tp0.1: 11303319.0000 - fp0.1: 5310555.0000 - tn0.1: 61362872.0000 - fn0.1: 666448.0000 - precision0.1: 0.6804 -
    recall0.1: 0.9443 - tp0.3: 10769572.0000 - fp0.3: 2338920.0000 - tn0.3: 64334512.0000 - fn0.3: 1200195.0000 - precision0.3: 0.8216 - recall0.3: 0.8997 - tp0.5: 10293662.0000 - fp0.5: 13284
    81.0000 - tn0.5: 65344964.0000 - fn0.5: 1676105.0000 - precision0.5: 0.8857 - recall0.5: 0.8600 - tp0.7: 9608679.0000 - fp0.7: 652337.0000 - tn0.7: 66021108.0000 - fn0.7: 2361088.0000 - pr
    ecision0.7: 0.9364 - recall0.7: 0.8027 - tp0.9: 8281304.0000 - fp0.9: 189171.0000 - tn0.9: 66484272.0000 - fn0.9: 3688463.0000 - precision0.9: 0.9777 - recall0.9: 0.6919 - accuracy: 0.9618
     - auc: 0.9660 - f1: 0.2642 - val_loss: 0.3489 - val_tp0.1: 3071912.0000 - val_fp0.1: 2477377.0000 - val_tn0.1: 14020569.0000 - val_fn0.1: 90942.0000 - val_precision0.1: 0.5536 - val_recal
    l0.1: 0.9712 - val_tp0.3: 2983731.0000 - val_fp0.3: 1182747.0000 - val_tn0.3: 15315199.0000 - val_fn0.3: 179123.0000 - val_precision0.3: 0.7161 - val_recall0.3: 0.9434 - val_tp0.5: 2894503
    .0000 - val_fp0.5: 758141.0000 - val_tn0.5: 15739805.0000 - val_fn0.5: 268351.0000 - val_precision0.5: 0.7924 - val_recall0.5: 0.9152 - val_tp0.7: 2765944.0000 - val_fp0.7: 443418.0000 - v
    al_tn0.7: 16054528.0000 - val_fn0.7: 396910.0000 - val_precision0.7: 0.8618 - val_recall0.7: 0.8745 - val_tp0.9: 2524618.0000 - val_fp0.9: 206040.0000 - val_tn0.9: 16291906.0000 - val_fn0.
    9: 638236.0000 - val_precision0.9: 0.9245 - val_recall0.9: 0.7982 - val_accuracy: 0.9478 - val_auc: 0.9752 - val_f1: 0.2772
    Epoch 15/20
    240/240 [==============================] - 124s 516ms/step - loss: 0.2859 - tp0.1: 11302324.0000 - fp0.1: 4990158.0000 - tn0.1: 61683280.0000 - fn0.1: 667443.0000 - precision0.1: 0.6937 -
    recall0.1: 0.9442 - tp0.3: 10801269.0000 - fp0.3: 2296897.0000 - tn0.3: 64376528.0000 - fn0.3: 1168498.0000 - precision0.3: 0.8246 - recall0.3: 0.9024 - tp0.5: 10322016.0000 - fp0.5: 12928
    01.0000 - tn0.5: 65380632.0000 - fn0.5: 1647751.0000 - precision0.5: 0.8887 - recall0.5: 0.8623 - tp0.7: 9638120.0000 - fp0.7: 622814.0000 - tn0.7: 66050612.0000 - fn0.7: 2331647.0000 - pr
    ecision0.7: 0.9393 - recall0.7: 0.8052 - tp0.9: 8333108.0000 - fp0.9: 176077.0000 - tn0.9: 66497368.0000 - fn0.9: 3636659.0000 - precision0.9: 0.9793 - recall0.9: 0.6962 - accuracy: 0.9626
     - auc: 0.9664 - f1: 0.2642 - val_loss: 0.2609 - val_tp0.1: 3046775.0000 - val_fp0.1: 1468094.0000 - val_tn0.1: 15029852.0000 - val_fn0.1: 116079.0000 - val_precision0.1: 0.6748 - val_reca
    ll0.1: 0.9633 - val_tp0.3: 2926133.0000 - val_fp0.3: 616330.0000 - val_tn0.3: 15881616.0000 - val_fn0.3: 236721.0000 - val_precision0.3: 0.8260 - val_recall0.3: 0.9252 - val_tp0.5: 2805534
    .0000 - val_fp0.5: 344220.0000 - val_tn0.5: 16153726.0000 - val_fn0.5: 357320.0000 - val_precision0.5: 0.8907 - val_recall0.5: 0.8870 - val_tp0.7: 2613829.0000 - val_fp0.7: 161631.0000 - v
    al_tn0.7: 16336315.0000 - val_fn0.7: 549025.0000 - val_precision0.7: 0.9418 - val_recall0.7: 0.8264 - val_tp0.9: 2262420.0000 - val_fp0.9: 50786.0000 - val_tn0.9: 16447160.0000 - val_fn0.9
    : 900434.0000 - val_precision0.9: 0.9780 - val_recall0.9: 0.7153 - val_accuracy: 0.9643 - val_auc: 0.9756 - val_f1: 0.2772
    Epoch 16/20
    240/240 [==============================] - 124s 518ms/step - loss: 0.2760 - tp0.1: 11343439.0000 - fp0.1: 5034724.0000 - tn0.1: 61638692.0000 - fn0.1: 626328.0000 - precision0.1: 0.6926 -
    recall0.1: 0.9477 - tp0.3: 10839346.0000 - fp0.3: 2204383.0000 - tn0.3: 64469040.0000 - fn0.3: 1130421.0000 - precision0.3: 0.8310 - recall0.3: 0.9056 - tp0.5: 10384981.0000 - fp0.5: 12476
    08.0000 - tn0.5: 65425836.0000 - fn0.5: 1584786.0000 - precision0.5: 0.8927 - recall0.5: 0.8676 - tp0.7: 9712329.0000 - fp0.7: 596551.0000 - tn0.7: 66076892.0000 - fn0.7: 2257438.0000 - pr
    ecision0.7: 0.9421 - recall0.7: 0.8114 - tp0.9: 8432551.0000 - fp0.9: 162062.0000 - tn0.9: 66511360.0000 - fn0.9: 3537216.0000 - precision0.9: 0.9811 - recall0.9: 0.7045 - accuracy: 0.9640
     - auc: 0.9683 - f1: 0.2642 - val_loss: 0.2563 - val_tp0.1: 3029508.0000 - val_fp0.1: 1192309.0000 - val_tn0.1: 15305637.0000 - val_fn0.1: 133346.0000 - val_precision0.1: 0.7176 - val_reca
    ll0.1: 0.9578 - val_tp0.3: 2908219.0000 - val_fp0.3: 564874.0000 - val_tn0.3: 15933072.0000 - val_fn0.3: 254635.0000 - val_precision0.3: 0.8374 - val_recall0.3: 0.9195 - val_tp0.5: 2799190
    .0000 - val_fp0.5: 352643.0000 - val_tn0.5: 16145303.0000 - val_fn0.5: 363664.0000 - val_precision0.5: 0.8881 - val_recall0.5: 0.8850 - val_tp0.7: 2636887.0000 - val_fp0.7: 194123.0000 - v
    al_tn0.7: 16303823.0000 - val_fn0.7: 525967.0000 - val_precision0.7: 0.9314 - val_recall0.7: 0.8337 - val_tp0.9: 2320546.0000 - val_fp0.9: 72127.0000 - val_tn0.9: 16425819.0000 - val_fn0.9
    : 842308.0000 - val_precision0.9: 0.9699 - val_recall0.9: 0.7337 - val_accuracy: 0.9636 - val_auc: 0.9731 - val_f1: 0.2772
    Epoch 17/20
    240/240 [==============================] - 124s 518ms/step - loss: 0.2703 - tp0.1: 11357057.0000 - fp0.1: 4893464.0000 - tn0.1: 61779992.0000 - fn0.1: 612710.0000 - precision0.1: 0.6989 -
    recall0.1: 0.9488 - tp0.3: 10875567.0000 - fp0.3: 2221183.0000 - tn0.3: 64452244.0000 - fn0.3: 1094200.0000 - precision0.3: 0.8304 - recall0.3: 0.9086 - tp0.5: 10410193.0000 - fp0.5: 12391
    75.0000 - tn0.5: 65434264.0000 - fn0.5: 1559574.0000 - precision0.5: 0.8936 - recall0.5: 0.8697 - tp0.7: 9743863.0000 - fp0.7: 592187.0000 - tn0.7: 66081244.0000 - fn0.7: 2225904.0000 - pr
    ecision0.7: 0.9427 - recall0.7: 0.8140 - tp0.9: 8521739.0000 - fp0.9: 167850.0000 - tn0.9: 66505612.0000 - fn0.9: 3448028.0000 - precision0.9: 0.9807 - recall0.9: 0.7119 - accuracy: 0.9644
     - auc: 0.9690 - f1: 0.2642 - val_loss: 0.2701 - val_tp0.1: 3061862.0000 - val_fp0.1: 1755743.0000 - val_tn0.1: 14742203.0000 - val_fn0.1: 100992.0000 - val_precision0.1: 0.6356 - val_reca
    ll0.1: 0.9681 - val_tp0.3: 2945795.0000 - val_fp0.3: 710061.0000 - val_tn0.3: 15787885.0000 - val_fn0.3: 217059.0000 - val_precision0.3: 0.8058 - val_recall0.3: 0.9314 - val_tp0.5: 2833502
    .0000 - val_fp0.5: 394281.0000 - val_tn0.5: 16103665.0000 - val_fn0.5: 329352.0000 - val_precision0.5: 0.8778 - val_recall0.5: 0.8959 - val_tp0.7: 2650030.0000 - val_fp0.7: 181493.0000 - v
    al_tn0.7: 16316453.0000 - val_fn0.7: 512824.0000 - val_precision0.7: 0.9359 - val_recall0.7: 0.8379 - val_tp0.9: 2293107.0000 - val_fp0.9: 49911.0000 - val_tn0.9: 16448035.0000 - val_fn0.9
    : 869747.0000 - val_precision0.9: 0.9787 - val_recall0.9: 0.7250 - val_accuracy: 0.9632 - val_auc: 0.9773 - val_f1: 0.2772
    Epoch 18/20
    240/240 [==============================] - 124s 518ms/step - loss: 0.2628 - tp0.1: 11372952.0000 - fp0.1: 4780985.0000 - tn0.1: 61892464.0000 - fn0.1: 596815.0000 - precision0.1: 0.7040 -
    recall0.1: 0.9501 - tp0.3: 10899958.0000 - fp0.3: 2139815.0000 - tn0.3: 64533632.0000 - fn0.3: 1069809.0000 - precision0.3: 0.8359 - recall0.3: 0.9106 - tp0.5: 10471625.0000 - fp0.5: 12249
    84.0000 - tn0.5: 65448432.0000 - fn0.5: 1498142.0000 - precision0.5: 0.8953 - recall0.5: 0.8748 - tp0.7: 9815694.0000 - fp0.7: 581437.0000 - tn0.7: 66091992.0000 - fn0.7: 2154073.0000 - pr
    ecision0.7: 0.9441 - recall0.7: 0.8200 - tp0.9: 8590979.0000 - fp0.9: 158690.0000 - tn0.9: 66514760.0000 - fn0.9: 3378788.0000 - precision0.9: 0.9819 - recall0.9: 0.7177 - accuracy: 0.9654
     - auc: 0.9700 - f1: 0.2642 - val_loss: 0.2858 - val_tp0.1: 3079296.0000 - val_fp0.1: 2141364.0000 - val_tn0.1: 14356582.0000 - val_fn0.1: 83558.0000 - val_precision0.1: 0.5898 - val_recal
    l0.1: 0.9736 - val_tp0.3: 2982065.0000 - val_fp0.3: 935179.0000 - val_tn0.3: 15562767.0000 - val_fn0.3: 180789.0000 - val_precision0.3: 0.7613 - val_recall0.3: 0.9428 - val_tp0.5: 2881468.
    0000 - val_fp0.5: 536262.0000 - val_tn0.5: 15961684.0000 - val_fn0.5: 281386.0000 - val_precision0.5: 0.8431 - val_recall0.5: 0.9110 - val_tp0.7: 2708837.0000 - val_fp0.7: 237110.0000 - va
    l_tn0.7: 16260836.0000 - val_fn0.7: 454017.0000 - val_precision0.7: 0.9195 - val_recall0.7: 0.8565 - val_tp0.9: 2296651.0000 - val_fp0.9: 53102.0000 - val_tn0.9: 16444844.0000 - val_fn0.9:
     866203.0000 - val_precision0.9: 0.9774 - val_recall0.9: 0.7261 - val_accuracy: 0.9584 - val_auc: 0.9793 - val_f1: 0.2772
    Epoch 19/20
    240/240 [==============================] - 124s 516ms/step - loss: 0.2557 - tp0.1: 11390165.0000 - fp0.1: 4618150.0000 - tn0.1: 62055280.0000 - fn0.1: 579602.0000 - precision0.1: 0.7115 -
    recall0.1: 0.9516 - tp0.3: 10925273.0000 - fp0.3: 2091134.0000 - tn0.3: 64582276.0000 - fn0.3: 1044494.0000 - precision0.3: 0.8393 - recall0.3: 0.9127 - tp0.5: 10488558.0000 - fp0.5: 11931
    41.0000 - tn0.5: 65480284.0000 - fn0.5: 1481209.0000 - precision0.5: 0.8979 - recall0.5: 0.8763 - tp0.7: 9830525.0000 - fp0.7: 561047.0000 - tn0.7: 66112360.0000 - fn0.7: 2139242.0000 - pr
    ecision0.7: 0.9460 - recall0.7: 0.8213 - tp0.9: 8625871.0000 - fp0.9: 152724.0000 - tn0.9: 66520724.0000 - fn0.9: 3343896.0000 - precision0.9: 0.9826 - recall0.9: 0.7206 - accuracy: 0.9660
     - auc: 0.9711 - f1: 0.2642 - val_loss: 0.2528 - val_tp0.1: 3043668.0000 - val_fp0.1: 1335624.0000 - val_tn0.1: 15162322.0000 - val_fn0.1: 119186.0000 - val_precision0.1: 0.6950 - val_reca
    ll0.1: 0.9623 - val_tp0.3: 2935979.0000 - val_fp0.3: 619716.0000 - val_tn0.3: 15878230.0000 - val_fn0.3: 226875.0000 - val_precision0.3: 0.8257 - val_recall0.3: 0.9283 - val_tp0.5: 2834409
    .0000 - val_fp0.5: 367871.0000 - val_tn0.5: 16130075.0000 - val_fn0.5: 328445.0000 - val_precision0.5: 0.8851 - val_recall0.5: 0.8962 - val_tp0.7: 2670269.0000 - val_fp0.7: 182097.0000 - v
    al_tn0.7: 16315849.0000 - val_fn0.7: 492585.0000 - val_precision0.7: 0.9362 - val_recall0.7: 0.8443 - val_tp0.9: 2324993.0000 - val_fp0.9: 54375.0000 - val_tn0.9: 16443571.0000 - val_fn0.9
    : 837861.0000 - val_precision0.9: 0.9771 - val_recall0.9: 0.7351 - val_accuracy: 0.9646 - val_auc: 0.9758 - val_f1: 0.2772
    Epoch 20/20
    240/240 [==============================] - 124s 518ms/step - loss: 0.2510 - tp0.1: 11400658.0000 - fp0.1: 4532014.0000 - tn0.1: 62141432.0000 - fn0.1: 569109.0000 - precision0.1: 0.7156 -
    recall0.1: 0.9525 - tp0.3: 10961511.0000 - fp0.3: 2108269.0000 - tn0.3: 64565152.0000 - fn0.3: 1008256.0000 - precision0.3: 0.8387 - recall0.3: 0.9158 - tp0.5: 10542090.0000 - fp0.5: 12141
    91.0000 - tn0.5: 65459224.0000 - fn0.5: 1427677.0000 - precision0.5: 0.8967 - recall0.5: 0.8807 - tp0.7: 9900824.0000 - fp0.7: 577656.0000 - tn0.7: 66095784.0000 - fn0.7: 2068943.0000 - pr
    ecision0.7: 0.9449 - recall0.7: 0.8272 - tp0.9: 8716202.0000 - fp0.9: 157381.0000 - tn0.9: 66516056.0000 - fn0.9: 3253565.0000 - precision0.9: 0.9823 - recall0.9: 0.7282 - accuracy: 0.9664
     - auc: 0.9716 - f1: 0.2642 - val_loss: 0.2602 - val_tp0.1: 3022436.0000 - val_fp0.1: 1215092.0000 - val_tn0.1: 15282854.0000 - val_fn0.1: 140418.0000 - val_precision0.1: 0.7133 - val_reca
    ll0.1: 0.9556 - val_tp0.3: 2889313.0000 - val_fp0.3: 511507.0000 - val_tn0.3: 15986439.0000 - val_fn0.3: 273541.0000 - val_precision0.3: 0.8496 - val_recall0.3: 0.9135 - val_tp0.5: 2761628
    .0000 - val_fp0.5: 283172.0000 - val_tn0.5: 16214774.0000 - val_fn0.5: 401226.0000 - val_precision0.5: 0.9070 - val_recall0.5: 0.8731 - val_tp0.7: 2568361.0000 - val_fp0.7: 130133.0000 - v
    al_tn0.7: 16367813.0000 - val_fn0.7: 594493.0000 - val_precision0.7: 0.9518 - val_recall0.7: 0.8120 - val_tp0.9: 2202966.0000 - val_fp0.9: 36402.0000 - val_tn0.9: 16461544.0000 - val_fn0.9
    : 959888.0000 - val_precision0.9: 0.9837 - val_recall0.9: 0.6965 - val_accuracy: 0.9652 - val_auc: 0.9728 - val_f1: 0.2772
    --- Running training session 63/140
    {'hp_epochs': 20, 'hp_batch_size': 14, 'hp_scaler': 'quant_g', 'hp_n_levels': 7, 'hp_first_filters': 16, 'hp_pool_size': 2, 'hp_input_size': 4096, 'hp_lr_start': 0.027144629354532844, 'hp_
    lr_power': 5.0}
    --- repeat #: 1
    input - shape:   (None, 4096, 1)
    output - shape:  (None, 4096, 1)
    Epoch 1/20
    2021-08-08 13:46:55.765737: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:177] Filling up shuffle buffer (this may take a while): 3624 of 4800
    2021-08-08 13:46:58.998412: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:230] Shuffle buffer filled.
    342/342 [==============================] - 60s 96ms/step - loss: 0.9388 - tp0.1: 2551056.0000 - fp0.1: 5602226.0000 - tn0.1: 11036673.0000 - fn0.1: 421693.0000 - precision0.1: 0.3129 - rec
    all0.1: 0.8581 - tp0.3: 1927079.0000 - fp0.3: 2492968.0000 - tn0.3: 14145931.0000 - fn0.3: 1045670.0000 - precision0.3: 0.4360 - recall0.3: 0.6482 - tp0.5: 1246353.0000 - fp0.5: 839125.000
    0 - tn0.5: 15799774.0000 - fn0.5: 1726396.0000 - precision0.5: 0.5976 - recall0.5: 0.4193 - tp0.7: 687790.0000 - fp0.7: 234530.0000 - tn0.7: 16404369.0000 - fn0.7: 2284959.0000 - precision
    0.7: 0.7457 - recall0.7: 0.2314 - tp0.9: 115373.0000 - fp0.9: 19205.0000 - tn0.9: 16619694.0000 - fn0.9: 2857376.0000 - precision0.9: 0.8573 - recall0.9: 0.0388 - accuracy: 0.8692 - auc: 0
    .8319 - f1: 0.2633 - val_loss: 1.2163 - val_tp0.1: 649575.0000 - val_fp0.1: 1129986.0000 - val_tn0.1: 2947304.0000 - val_fn0.1: 147375.0000 - val_precision0.1: 0.3650 - val_recall0.1: 0.81
    51 - val_tp0.3: 600297.0000 - val_fp0.3: 897583.0000 - val_tn0.3: 3179707.0000 - val_fn0.3: 196653.0000 - val_precision0.3: 0.4008 - val_recall0.3: 0.7532 - val_tp0.5: 562997.0000 - val_fp
    0.5: 731818.0000 - val_tn0.5: 3345472.0000 - val_fn0.5: 233953.0000 - val_precision0.5: 0.4348 - val_recall0.5: 0.7064 - val_tp0.7: 521584.0000 - val_fp0.7: 583432.0000 - val_tn0.7: 349385
    8.0000 - val_fn0.7: 275366.0000 - val_precision0.7: 0.4720 - val_recall0.7: 0.6545 - val_tp0.9: 432196.0000 - val_fp0.9: 357157.0000 - val_tn0.9: 3720133.0000 - val_fn0.9: 364754.0000 - va
    l_precision0.9: 0.5475 - val_recall0.9: 0.5423 - val_accuracy: 0.8019 - val_auc: 0.8254 - val_f1: 0.2811
    Epoch 2/20
    342/342 [==============================] - 27s 79ms/step - loss: 0.8637 - tp0.1: 2563159.0000 - fp0.1: 5123938.0000 - tn0.1: 11520950.0000 - fn0.1: 403601.0000 - precision0.1: 0.3334 - rec
    all0.1: 0.8640 - tp0.3: 2082014.0000 - fp0.3: 2404817.0000 - tn0.3: 14240071.0000 - fn0.3: 884746.0000 - precision0.3: 0.4640 - recall0.3: 0.7018 - tp0.5: 1378923.0000 - fp0.5: 865967.0000
     - tn0.5: 15778921.0000 - fn0.5: 1587837.0000 - precision0.5: 0.6142 - recall0.5: 0.4648 - tp0.7: 747621.0000 - fp0.7: 249304.0000 - tn0.7: 16395584.0000 - fn0.7: 2219139.0000 - precision0
    .7: 0.7499 - recall0.7: 0.2520 - tp0.9: 156922.0000 - fp0.9: 25309.0000 - tn0.9: 16619579.0000 - fn0.9: 2809838.0000 - precision0.9: 0.8611 - recall0.9: 0.0529 - accuracy: 0.8749 - auc: 0.
    8499 - f1: 0.2628 - val_loss: 6.7415 - val_tp0.1: 794760.0000 - val_fp0.1: 3364179.0000 - val_tn0.1: 712620.0000 - val_fn0.1: 2681.0000 - val_precision0.1: 0.1911 - val_recall0.1: 0.9966 -
     val_tp0.3: 790967.0000 - val_fp0.3: 2758328.0000 - val_tn0.3: 1318471.0000 - val_fn0.3: 6474.0000 - val_precision0.3: 0.2229 - val_recall0.3: 0.9919 - val_tp0.5: 786264.0000 - val_fp0.5:
    2520461.0000 - val_tn0.5: 1556338.0000 - val_fn0.5: 11177.0000 - val_precision0.5: 0.2378 - val_recall0.5: 0.9860 - val_tp0.7: 779634.0000 - val_fp0.7: 2343349.0000 - val_tn0.7: 1733450.00
    00 - val_fn0.7: 17807.0000 - val_precision0.7: 0.2496 - val_recall0.7: 0.9777 - val_tp0.9: 763809.0000 - val_fp0.9: 2124191.0000 - val_tn0.9: 1952608.0000 - val_fn0.9: 33632.0000 - val_pre
    cision0.9: 0.2645 - val_recall0.9: 0.9578 - val_accuracy: 0.4806 - val_auc: 0.7467 - val_f1: 0.2812
    Epoch 3/20
    342/342 [==============================] - 30s 88ms/step - loss: 0.7536 - tp0.1: 2653716.0000 - fp0.1: 4711992.0000 - tn0.1: 11929767.0000 - fn0.1: 316173.0000 - precision0.1: 0.3603 - rec
    all0.1: 0.8935 - tp0.3: 2203201.0000 - fp0.3: 2044395.0000 - tn0.3: 14597364.0000 - fn0.3: 766688.0000 - precision0.3: 0.5187 - recall0.3: 0.7418 - tp0.5: 1692995.0000 - fp0.5: 943961.0000
     - tn0.5: 15697798.0000 - fn0.5: 1276894.0000 - precision0.5: 0.6420 - recall0.5: 0.5701 - tp0.7: 1039318.0000 - fp0.7: 338210.0000 - tn0.7: 16303549.0000 - fn0.7: 1930571.0000 - precision
    0.7: 0.7545 - recall0.7: 0.3500 - tp0.9: 234516.0000 - fp0.9: 36668.0000 - tn0.9: 16605091.0000 - fn0.9: 2735373.0000 - precision0.9: 0.8648 - recall0.9: 0.0790 - accuracy: 0.8868 - auc: 0
    .8820 - f1: 0.2630 - val_loss: 1.5030 - val_tp0.1: 736916.0000 - val_fp0.1: 1516908.0000 - val_tn0.1: 2557738.0000 - val_fn0.1: 62678.0000 - val_precision0.1: 0.3270 - val_recall0.1: 0.921
    6 - val_tp0.3: 720401.0000 - val_fp0.3: 1289822.0000 - val_tn0.3: 2784824.0000 - val_fn0.3: 79193.0000 - val_precision0.3: 0.3584 - val_recall0.3: 0.9010 - val_tp0.5: 705808.0000 - val_fp0
    .5: 1149637.0000 - val_tn0.5: 2925009.0000 - val_fn0.5: 93786.0000 - val_precision0.5: 0.3804 - val_recall0.5: 0.8827 - val_tp0.7: 685398.0000 - val_fp0.7: 986538.0000 - val_tn0.7: 3088108
    .0000 - val_fn0.7: 114196.0000 - val_precision0.7: 0.4099 - val_recall0.7: 0.8572 - val_tp0.9: 644148.0000 - val_fp0.9: 756371.0000 - val_tn0.9: 3318275.0000 - val_fn0.9: 155446.0000 - val
    _precision0.9: 0.4599 - val_recall0.9: 0.8056 - val_accuracy: 0.7449 - val_auc: 0.8651 - val_f1: 0.2819
    Epoch 4/20
    342/342 [==============================] - 30s 88ms/step - loss: 0.6853 - tp0.1: 2703897.0000 - fp0.1: 4342508.0000 - tn0.1: 12288832.0000 - fn0.1: 276411.0000 - precision0.1: 0.3837 - rec
    all0.1: 0.9073 - tp0.3: 2293716.0000 - fp0.3: 1900715.0000 - tn0.3: 14730625.0000 - fn0.3: 686592.0000 - precision0.3: 0.5468 - recall0.3: 0.7696 - tp0.5: 1860758.0000 - fp0.5: 965942.0000
     - tn0.5: 15665398.0000 - fn0.5: 1119550.0000 - precision0.5: 0.6583 - recall0.5: 0.6244 - tp0.7: 1232630.0000 - fp0.7: 369641.0000 - tn0.7: 16261699.0000 - fn0.7: 1747678.0000 - precision
    0.7: 0.7693 - recall0.7: 0.4136 - tp0.9: 331288.0000 - fp0.9: 50757.0000 - tn0.9: 16580583.0000 - fn0.9: 2649020.0000 - precision0.9: 0.8671 - recall0.9: 0.1112 - accuracy: 0.8937 - auc: 0
    .8982 - f1: 0.2638 - val_loss: 0.9547 - val_tp0.1: 735656.0000 - val_fp0.1: 1474945.0000 - val_tn0.1: 2608195.0000 - val_fn0.1: 55444.0000 - val_precision0.1: 0.3328 - val_recall0.1: 0.929
    9 - val_tp0.3: 692992.0000 - val_fp0.3: 1096764.0000 - val_tn0.3: 2986376.0000 - val_fn0.3: 98108.0000 - val_precision0.3: 0.3872 - val_recall0.3: 0.8760 - val_tp0.5: 651741.0000 - val_fp0
    .5: 858021.0000 - val_tn0.5: 3225119.0000 - val_fn0.5: 139359.0000 - val_precision0.5: 0.4317 - val_recall0.5: 0.8238 - val_tp0.7: 578859.0000 - val_fp0.7: 602934.0000 - val_tn0.7: 3480206
    .0000 - val_fn0.7: 212241.0000 - val_precision0.7: 0.4898 - val_recall0.7: 0.7317 - val_tp0.9: 421648.0000 - val_fp0.9: 292420.0000 - val_tn0.9: 3790720.0000 - val_fn0.9: 369452.0000 - val
    _precision0.9: 0.5905 - val_recall0.9: 0.5330 - val_accuracy: 0.7954 - val_auc: 0.8742 - val_f1: 0.2793
    Epoch 5/20
    342/342 [==============================] - 30s 89ms/step - loss: 0.6168 - tp0.1: 2714795.0000 - fp0.1: 3825827.0000 - tn0.1: 12828162.0000 - fn0.1: 242864.0000 - precision0.1: 0.4151 - rec
    all0.1: 0.9179 - tp0.3: 2360594.0000 - fp0.3: 1759953.0000 - tn0.3: 14894036.0000 - fn0.3: 597065.0000 - precision0.3: 0.5729 - recall0.3: 0.7981 - tp0.5: 1993009.0000 - fp0.5: 943846.0000
     - tn0.5: 15710143.0000 - fn0.5: 964650.0000 - precision0.5: 0.6786 - recall0.5: 0.6738 - tp0.7: 1426514.0000 - fp0.7: 393147.0000 - tn0.7: 16260842.0000 - fn0.7: 1531145.0000 - precision0
    .7: 0.7839 - recall0.7: 0.4823 - tp0.9: 433784.0000 - fp0.9: 59911.0000 - tn0.9: 16594078.0000 - fn0.9: 2523875.0000 - precision0.9: 0.8786 - recall0.9: 0.1467 - accuracy: 0.9027 - auc: 0.
    9117 - f1: 0.2621 - val_loss: 0.6248 - val_tp0.1: 740961.0000 - val_fp0.1: 952224.0000 - val_tn0.1: 3122967.0000 - val_fn0.1: 58088.0000 - val_precision0.1: 0.4376 - val_recall0.1: 0.9273
    - val_tp0.3: 664317.0000 - val_fp0.3: 488340.0000 - val_tn0.3: 3586851.0000 - val_fn0.3: 134732.0000 - val_precision0.3: 0.5763 - val_recall0.3: 0.8314 - val_tp0.5: 563011.0000 - val_fp0.5
    : 245735.0000 - val_tn0.5: 3829456.0000 - val_fn0.5: 236038.0000 - val_precision0.5: 0.6962 - val_recall0.5: 0.7046 - val_tp0.7: 395949.0000 - val_fp0.7: 88632.0000 - val_tn0.7: 3986559.00
    00 - val_fn0.7: 403100.0000 - val_precision0.7: 0.8171 - val_recall0.7: 0.4955 - val_tp0.9: 112039.0000 - val_fp0.9: 5626.0000 - val_tn0.9: 4069565.0000 - val_fn0.9: 687010.0000 - val_prec
    ision0.9: 0.9522 - val_recall0.9: 0.1402 - val_accuracy: 0.9012 - val_auc: 0.9231 - val_f1: 0.2817
    Epoch 6/20
    342/342 [==============================] - 29s 84ms/step - loss: 0.5810 - tp0.1: 2723990.0000 - fp0.1: 3427161.0000 - tn0.1: 13216327.0000 - fn0.1: 244170.0000 - precision0.1: 0.4428 - rec
    all0.1: 0.9177 - tp0.3: 2397859.0000 - fp0.3: 1591770.0000 - tn0.3: 15051718.0000 - fn0.3: 570301.0000 - precision0.3: 0.6010 - recall0.3: 0.8079 - tp0.5: 2062821.0000 - fp0.5: 894260.0000
     - tn0.5: 15749228.0000 - fn0.5: 905339.0000 - precision0.5: 0.6976 - recall0.5: 0.6950 - tp0.7: 1513878.0000 - fp0.7: 385604.0000 - tn0.7: 16257884.0000 - fn0.7: 1454282.0000 - precision0
    .7: 0.7970 - recall0.7: 0.5100 - tp0.9: 521178.0000 - fp0.9: 63769.0000 - tn0.9: 16579719.0000 - fn0.9: 2446982.0000 - precision0.9: 0.8910 - recall0.9: 0.1756 - accuracy: 0.9082 - auc: 0.
    9187 - f1: 0.2629 - val_loss: 0.5978 - val_tp0.1: 736458.0000 - val_fp0.1: 812942.0000 - val_tn0.1: 3258937.0000 - val_fn0.1: 65903.0000 - val_precision0.1: 0.4753 - val_recall0.1: 0.9179
    - val_tp0.3: 673047.0000 - val_fp0.3: 451203.0000 - val_tn0.3: 3620676.0000 - val_fn0.3: 129314.0000 - val_precision0.3: 0.5987 - val_recall0.3: 0.8388 - val_tp0.5: 622273.0000 - val_fp0.5
    : 311162.0000 - val_tn0.5: 3760717.0000 - val_fn0.5: 180088.0000 - val_precision0.5: 0.6666 - val_recall0.5: 0.7756 - val_tp0.7: 543103.0000 - val_fp0.7: 183256.0000 - val_tn0.7: 3888623.0
    000 - val_fn0.7: 259258.0000 - val_precision0.7: 0.7477 - val_recall0.7: 0.6769 - val_tp0.9: 367886.0000 - val_fp0.9: 60748.0000 - val_tn0.9: 4011131.0000 - val_fn0.9: 434475.0000 - val_pr
    ecision0.9: 0.8583 - val_recall0.9: 0.4585 - val_accuracy: 0.8992 - val_auc: 0.9218 - val_f1: 0.2827
    Epoch 7/20
    342/342 [==============================] - 30s 87ms/step - loss: 0.5618 - tp0.1: 2753144.0000 - fp0.1: 3391488.0000 - tn0.1: 13244099.0000 - fn0.1: 222917.0000 - precision0.1: 0.4481 - rec
    all0.1: 0.9251 - tp0.3: 2430470.0000 - fp0.3: 1601144.0000 - tn0.3: 15034443.0000 - fn0.3: 545591.0000 - precision0.3: 0.6029 - recall0.3: 0.8167 - tp0.5: 2105059.0000 - fp0.5: 906072.0000
     - tn0.5: 15729515.0000 - fn0.5: 871002.0000 - precision0.5: 0.6991 - recall0.5: 0.7073 - tp0.7: 1566971.0000 - fp0.7: 391164.0000 - tn0.7: 16244423.0000 - fn0.7: 1409090.0000 - precision0
    .7: 0.8002 - recall0.7: 0.5265 - tp0.9: 568695.0000 - fp0.9: 65221.0000 - tn0.9: 16570366.0000 - fn0.9: 2407366.0000 - precision0.9: 0.8971 - recall0.9: 0.1911 - accuracy: 0.9094 - auc: 0.
    9229 - f1: 0.2635 - val_loss: 0.5711 - val_tp0.1: 744203.0000 - val_fp0.1: 929725.0000 - val_tn0.1: 3153721.0000 - val_fn0.1: 46591.0000 - val_precision0.1: 0.4446 - val_recall0.1: 0.9411
    - val_tp0.3: 640265.0000 - val_fp0.3: 353301.0000 - val_tn0.3: 3730145.0000 - val_fn0.3: 150529.0000 - val_precision0.3: 0.6444 - val_recall0.3: 0.8096 - val_tp0.5: 490709.0000 - val_fp0.5
    : 135186.0000 - val_tn0.5: 3948260.0000 - val_fn0.5: 300085.0000 - val_precision0.5: 0.7840 - val_recall0.5: 0.6205 - val_tp0.7: 277364.0000 - val_fp0.7: 34487.0000 - val_tn0.7: 4048959.00
    00 - val_fn0.7: 513430.0000 - val_precision0.7: 0.8894 - val_recall0.7: 0.3507 - val_tp0.9: 24614.0000 - val_fp0.9: 1054.0000 - val_tn0.9: 4082392.0000 - val_fn0.9: 766180.0000 - val_preci
    sion0.9: 0.9589 - val_recall0.9: 0.0311 - val_accuracy: 0.9107 - val_auc: 0.9298 - val_f1: 0.2792
    Epoch 8/20
    342/342 [==============================] - 28s 81ms/step - loss: 0.5408 - tp0.1: 2752489.0000 - fp0.1: 3321723.0000 - tn0.1: 13326977.0000 - fn0.1: 210459.0000 - precision0.1: 0.4531 - rec
    all0.1: 0.9290 - tp0.3: 2438347.0000 - fp0.3: 1518540.0000 - tn0.3: 15130160.0000 - fn0.3: 524601.0000 - precision0.3: 0.6162 - recall0.3: 0.8229 - tp0.5: 2124376.0000 - fp0.5: 851682.0000
     - tn0.5: 15797018.0000 - fn0.5: 838572.0000 - precision0.5: 0.7138 - recall0.5: 0.7170 - tp0.7: 1623433.0000 - fp0.7: 378788.0000 - tn0.7: 16269912.0000 - fn0.7: 1339515.0000 - precision0
    .7: 0.8108 - recall0.7: 0.5479 - tp0.9: 664330.0000 - fp0.9: 71506.0000 - tn0.9: 16577194.0000 - fn0.9: 2298618.0000 - precision0.9: 0.9028 - recall0.9: 0.2242 - accuracy: 0.9138 - auc: 0.
    9275 - f1: 0.2625 - val_loss: 0.5804 - val_tp0.1: 719809.0000 - val_fp0.1: 677691.0000 - val_tn0.1: 3399808.0000 - val_fn0.1: 76932.0000 - val_precision0.1: 0.5151 - val_recall0.1: 0.9034
    - val_tp0.3: 629328.0000 - val_fp0.3: 315236.0000 - val_tn0.3: 3762263.0000 - val_fn0.3: 167413.0000 - val_precision0.3: 0.6663 - val_recall0.3: 0.7899 - val_tp0.5: 549596.0000 - val_fp0.5
    : 183412.0000 - val_tn0.5: 3894087.0000 - val_fn0.5: 247145.0000 - val_precision0.5: 0.7498 - val_recall0.5: 0.6898 - val_tp0.7: 420608.0000 - val_fp0.7: 82009.0000 - val_tn0.7: 3995490.00
    00 - val_fn0.7: 376133.0000 - val_precision0.7: 0.8368 - val_recall0.7: 0.5279 - val_tp0.9: 188872.0000 - val_fp0.9: 12738.0000 - val_tn0.9: 4064761.0000 - val_fn0.9: 607869.0000 - val_pre
    cision0.9: 0.9368 - val_recall0.9: 0.2371 - val_accuracy: 0.9117 - val_auc: 0.9182 - val_f1: 0.2810
    Epoch 9/20
    342/342 [==============================] - 31s 90ms/step - loss: 0.5325 - tp0.1: 2770579.0000 - fp0.1: 3319105.0000 - tn0.1: 13319456.0000 - fn0.1: 202508.0000 - precision0.1: 0.4550 - rec
    all0.1: 0.9319 - tp0.3: 2466041.0000 - fp0.3: 1533716.0000 - tn0.3: 15104845.0000 - fn0.3: 507046.0000 - precision0.3: 0.6165 - recall0.3: 0.8295 - tp0.5: 2177107.0000 - fp0.5: 886546.0000
     - tn0.5: 15752015.0000 - fn0.5: 795980.0000 - precision0.5: 0.7106 - recall0.5: 0.7323 - tp0.7: 1685024.0000 - fp0.7: 402104.0000 - tn0.7: 16236457.0000 - fn0.7: 1288063.0000 - precision0
    .7: 0.8073 - recall0.7: 0.5668 - tp0.9: 684970.0000 - fp0.9: 71002.0000 - tn0.9: 16567559.0000 - fn0.9: 2288117.0000 - precision0.9: 0.9061 - recall0.9: 0.2304 - accuracy: 0.9142 - auc: 0.
    9299 - f1: 0.2633 - val_loss: 0.6069 - val_tp0.1: 776068.0000 - val_fp0.1: 1351519.0000 - val_tn0.1: 2726452.0000 - val_fn0.1: 20201.0000 - val_precision0.1: 0.3648 - val_recall0.1: 0.9746
     - val_tp0.3: 720731.0000 - val_fp0.3: 628859.0000 - val_tn0.3: 3449112.0000 - val_fn0.3: 75538.0000 - val_precision0.3: 0.5340 - val_recall0.3: 0.9051 - val_tp0.5: 648641.0000 - val_fp0.5
    : 344147.0000 - val_tn0.5: 3733824.0000 - val_fn0.5: 147628.0000 - val_precision0.5: 0.6534 - val_recall0.5: 0.8146 - val_tp0.7: 538210.0000 - val_fp0.7: 168258.0000 - val_tn0.7: 3909713.0
    000 - val_fn0.7: 258059.0000 - val_precision0.7: 0.7618 - val_recall0.7: 0.6759 - val_tp0.9: 289556.0000 - val_fp0.9: 34074.0000 - val_tn0.9: 4043897.0000 - val_fn0.9: 506713.0000 - val_pr
    ecision0.9: 0.8947 - val_recall0.9: 0.3636 - val_accuracy: 0.8991 - val_auc: 0.9424 - val_f1: 0.2808
    Epoch 10/20
    342/342 [==============================] - 29s 86ms/step - loss: 0.5031 - tp0.1: 2780826.0000 - fp0.1: 3016769.0000 - tn0.1: 13618075.0000 - fn0.1: 195978.0000 - precision0.1: 0.4797 - rec
    all0.1: 0.9342 - tp0.3: 2488133.0000 - fp0.3: 1399371.0000 - tn0.3: 15235473.0000 - fn0.3: 488671.0000 - precision0.3: 0.6400 - recall0.3: 0.8358 - tp0.5: 2202797.0000 - fp0.5: 811436.0000
     - tn0.5: 15823408.0000 - fn0.5: 774007.0000 - precision0.5: 0.7308 - recall0.5: 0.7400 - tp0.7: 1728965.0000 - fp0.7: 377368.0000 - tn0.7: 16257476.0000 - fn0.7: 1247839.0000 - precision0
    .7: 0.8208 - recall0.7: 0.5808 - tp0.9: 720528.0000 - fp0.9: 68075.0000 - tn0.9: 16566769.0000 - fn0.9: 2256276.0000 - precision0.9: 0.9137 - recall0.9: 0.2420 - accuracy: 0.9192 - auc: 0.
    9351 - f1: 0.2636 - val_loss: 0.5387 - val_tp0.1: 760105.0000 - val_fp0.1: 938438.0000 - val_tn0.1: 3135229.0000 - val_fn0.1: 40468.0000 - val_precision0.1: 0.4475 - val_recall0.1: 0.9495
    - val_tp0.3: 691484.0000 - val_fp0.3: 448766.0000 - val_tn0.3: 3624901.0000 - val_fn0.3: 109089.0000 - val_precision0.3: 0.6064 - val_recall0.3: 0.8637 - val_tp0.5: 628663.0000 - val_fp0.5
    : 268056.0000 - val_tn0.5: 3805611.0000 - val_fn0.5: 171910.0000 - val_precision0.5: 0.7011 - val_recall0.5: 0.7853 - val_tp0.7: 517637.0000 - val_fp0.7: 128039.0000 - val_tn0.7: 3945628.0
    000 - val_fn0.7: 282936.0000 - val_precision0.7: 0.8017 - val_recall0.7: 0.6466 - val_tp0.9: 267423.0000 - val_fp0.9: 24096.0000 - val_tn0.9: 4049571.0000 - val_fn0.9: 533150.0000 - val_pr
    ecision0.9: 0.9173 - val_recall0.9: 0.3340 - val_accuracy: 0.9097 - val_auc: 0.9378 - val_f1: 0.2821
    Epoch 11/20
    342/342 [==============================] - 30s 88ms/step - loss: 0.5001 - tp0.1: 2760834.0000 - fp0.1: 2989115.0000 - tn0.1: 13662862.0000 - fn0.1: 198837.0000 - precision0.1: 0.4801 - rec
    all0.1: 0.9328 - tp0.3: 2478896.0000 - fp0.3: 1396965.0000 - tn0.3: 15255012.0000 - fn0.3: 480775.0000 - precision0.3: 0.6396 - recall0.3: 0.8376 - tp0.5: 2214110.0000 - fp0.5: 817049.0000
     - tn0.5: 15834928.0000 - fn0.5: 745561.0000 - precision0.5: 0.7304 - recall0.5: 0.7481 - tp0.7: 1770056.0000 - fp0.7: 384025.0000 - tn0.7: 16267952.0000 - fn0.7: 1189615.0000 - precision0
    .7: 0.8217 - recall0.7: 0.5981 - tp0.9: 798931.0000 - fp0.9: 75031.0000 - tn0.9: 16576946.0000 - fn0.9: 2160740.0000 - precision0.9: 0.9141 - recall0.9: 0.2699 - accuracy: 0.9203 - auc: 0.
    9351 - f1: 0.2623 - val_loss: 0.5461 - val_tp0.1: 730695.0000 - val_fp0.1: 658715.0000 - val_tn0.1: 3417454.0000 - val_fn0.1: 67376.0000 - val_precision0.1: 0.5259 - val_recall0.1: 0.9156
    - val_tp0.3: 621793.0000 - val_fp0.3: 261105.0000 - val_tn0.3: 3815064.0000 - val_fn0.3: 176278.0000 - val_precision0.3: 0.7043 - val_recall0.3: 0.7791 - val_tp0.5: 521488.0000 - val_fp0.5
    : 133143.0000 - val_tn0.5: 3943026.0000 - val_fn0.5: 276583.0000 - val_precision0.5: 0.7966 - val_recall0.5: 0.6534 - val_tp0.7: 362988.0000 - val_fp0.7: 49452.0000 - val_tn0.7: 4026717.00
    00 - val_fn0.7: 435083.0000 - val_precision0.7: 0.8801 - val_recall0.7: 0.4548 - val_tp0.9: 99821.0000 - val_fp0.9: 3904.0000 - val_tn0.9: 4072265.0000 - val_fn0.9: 698250.0000 - val_preci
    sion0.9: 0.9624 - val_recall0.9: 0.1251 - val_accuracy: 0.9159 - val_auc: 0.9274 - val_f1: 0.2814
    Epoch 12/20
    342/342 [==============================] - 30s 87ms/step - loss: 0.4965 - tp0.1: 2780843.0000 - fp0.1: 3042674.0000 - tn0.1: 13597784.0000 - fn0.1: 190347.0000 - precision0.1: 0.4775 - rec
    all0.1: 0.9359 - tp0.3: 2483369.0000 - fp0.3: 1382732.0000 - tn0.3: 15257726.0000 - fn0.3: 487821.0000 - precision0.3: 0.6423 - recall0.3: 0.8358 - tp0.5: 2206966.0000 - fp0.5: 794121.0000
     - tn0.5: 15846337.0000 - fn0.5: 764224.0000 - precision0.5: 0.7354 - recall0.5: 0.7428 - tp0.7: 1750172.0000 - fp0.7: 362712.0000 - tn0.7: 16277746.0000 - fn0.7: 1221018.0000 - precision0
    .7: 0.8283 - recall0.7: 0.5890 - tp0.9: 762690.0000 - fp0.9: 68788.0000 - tn0.9: 16571670.0000 - fn0.9: 2208500.0000 - precision0.9: 0.9173 - recall0.9: 0.2567 - accuracy: 0.9205 - auc: 0.
    9366 - f1: 0.2631 - val_loss: 0.5341 - val_tp0.1: 758000.0000 - val_fp0.1: 932695.0000 - val_tn0.1: 3144776.0000 - val_fn0.1: 38769.0000 - val_precision0.1: 0.4483 - val_recall0.1: 0.9513
    - val_tp0.3: 691823.0000 - val_fp0.3: 454271.0000 - val_tn0.3: 3623200.0000 - val_fn0.3: 104946.0000 - val_precision0.3: 0.6036 - val_recall0.3: 0.8683 - val_tp0.5: 633340.0000 - val_fp0.5
    : 278624.0000 - val_tn0.5: 3798847.0000 - val_fn0.5: 163429.0000 - val_precision0.5: 0.6945 - val_recall0.5: 0.7949 - val_tp0.7: 534109.0000 - val_fp0.7: 146703.0000 - val_tn0.7: 3930768.0
    000 - val_fn0.7: 262660.0000 - val_precision0.7: 0.7845 - val_recall0.7: 0.6703 - val_tp0.9: 318438.0000 - val_fp0.9: 38771.0000 - val_tn0.9: 4038700.0000 - val_fn0.9: 478331.0000 - val_pr
    ecision0.9: 0.8915 - val_recall0.9: 0.3997 - val_accuracy: 0.9093 - val_auc: 0.9388 - val_f1: 0.2810
    Epoch 13/20
    342/342 [==============================] - 29s 84ms/step - loss: 0.4892 - tp0.1: 2779063.0000 - fp0.1: 2979477.0000 - tn0.1: 13665550.0000 - fn0.1: 187558.0000 - precision0.1: 0.4826 - rec
    all0.1: 0.9368 - tp0.3: 2496191.0000 - fp0.3: 1375175.0000 - tn0.3: 15269852.0000 - fn0.3: 470430.0000 - precision0.3: 0.6448 - recall0.3: 0.8414 - tp0.5: 2236501.0000 - fp0.5: 810489.0000
     - tn0.5: 15834538.0000 - fn0.5: 730120.0000 - precision0.5: 0.7340 - recall0.5: 0.7539 - tp0.7: 1787849.0000 - fp0.7: 380585.0000 - tn0.7: 16264442.0000 - fn0.7: 1178772.0000 - precision0
    .7: 0.8245 - recall0.7: 0.6027 - tp0.9: 798725.0000 - fp0.9: 77694.0000 - tn0.9: 16567333.0000 - fn0.9: 2167896.0000 - precision0.9: 0.9114 - recall0.9: 0.2692 - accuracy: 0.9214 - auc: 0.
    9378 - f1: 0.2628 - val_loss: 0.5294 - val_tp0.1: 755023.0000 - val_fp0.1: 867873.0000 - val_tn0.1: 3206834.0000 - val_fn0.1: 44510.0000 - val_precision0.1: 0.4652 - val_recall0.1: 0.9443
    - val_tp0.3: 673836.0000 - val_fp0.3: 383349.0000 - val_tn0.3: 3691358.0000 - val_fn0.3: 125697.0000 - val_precision0.3: 0.6374 - val_recall0.3: 0.8428 - val_tp0.5: 597569.0000 - val_fp0.5
    : 210770.0000 - val_tn0.5: 3863937.0000 - val_fn0.5: 201964.0000 - val_precision0.5: 0.7393 - val_recall0.5: 0.7474 - val_tp0.7: 457652.0000 - val_fp0.7: 91160.0000 - val_tn0.7: 3983547.00
    00 - val_fn0.7: 341881.0000 - val_precision0.7: 0.8339 - val_recall0.7: 0.5724 - val_tp0.9: 170366.0000 - val_fp0.9: 11587.0000 - val_tn0.9: 4063120.0000 - val_fn0.9: 629167.0000 - val_pre
    cision0.9: 0.9363 - val_recall0.9: 0.2131 - val_accuracy: 0.9153 - val_auc: 0.9368 - val_f1: 0.2818
    Epoch 14/20
    342/342 [==============================] - 30s 87ms/step - loss: 0.4909 - tp0.1: 2796281.0000 - fp0.1: 2993242.0000 - tn0.1: 13633299.0000 - fn0.1: 188826.0000 - precision0.1: 0.4830 - rec
    all0.1: 0.9367 - tp0.3: 2509770.0000 - fp0.3: 1365834.0000 - tn0.3: 15260707.0000 - fn0.3: 475337.0000 - precision0.3: 0.6476 - recall0.3: 0.8408 - tp0.5: 2246662.0000 - fp0.5: 798278.0000
     - tn0.5: 15828263.0000 - fn0.5: 738445.0000 - precision0.5: 0.7378 - recall0.5: 0.7526 - tp0.7: 1791881.0000 - fp0.7: 369480.0000 - tn0.7: 16257061.0000 - fn0.7: 1193226.0000 - precision0
    .7: 0.8291 - recall0.7: 0.6003 - tp0.9: 775898.0000 - fp0.9: 73458.0000 - tn0.9: 16553083.0000 - fn0.9: 2209209.0000 - precision0.9: 0.9135 - recall0.9: 0.2599 - accuracy: 0.9216 - auc: 0.
    9379 - f1: 0.2642 - val_loss: 0.5282 - val_tp0.1: 755947.0000 - val_fp0.1: 899285.0000 - val_tn0.1: 3177862.0000 - val_fn0.1: 41146.0000 - val_precision0.1: 0.4567 - val_recall0.1: 0.9484
    - val_tp0.3: 680024.0000 - val_fp0.3: 410533.0000 - val_tn0.3: 3666614.0000 - val_fn0.3: 117069.0000 - val_precision0.3: 0.6236 - val_recall0.3: 0.8531 - val_tp0.5: 611482.0000 - val_fp0.5
    : 236410.0000 - val_tn0.5: 3840737.0000 - val_fn0.5: 185611.0000 - val_precision0.5: 0.7212 - val_recall0.5: 0.7671 - val_tp0.7: 485673.0000 - val_fp0.7: 108280.0000 - val_tn0.7: 3968867.0
    000 - val_fn0.7: 311420.0000 - val_precision0.7: 0.8177 - val_recall0.7: 0.6093 - val_tp0.9: 216751.0000 - val_fp0.9: 17398.0000 - val_tn0.9: 4059749.0000 - val_fn0.9: 580342.0000 - val_pr
    ecision0.9: 0.9257 - val_recall0.9: 0.2719 - val_accuracy: 0.9134 - val_auc: 0.9380 - val_f1: 0.2811
    Epoch 15/20
    342/342 [==============================] - 27s 80ms/step - loss: 0.4895 - tp0.1: 2778517.0000 - fp0.1: 2946222.0000 - tn0.1: 13695622.0000 - fn0.1: 191287.0000 - precision0.1: 0.4854 - rec
    all0.1: 0.9356 - tp0.3: 2488194.0000 - fp0.3: 1341585.0000 - tn0.3: 15300259.0000 - fn0.3: 481610.0000 - precision0.3: 0.6497 - recall0.3: 0.8378 - tp0.5: 2226779.0000 - fp0.5: 783374.0000
     - tn0.5: 15858470.0000 - fn0.5: 743025.0000 - precision0.5: 0.7398 - recall0.5: 0.7498 - tp0.7: 1768572.0000 - fp0.7: 363199.0000 - tn0.7: 16278645.0000 - fn0.7: 1201232.0000 - precision0
    .7: 0.8296 - recall0.7: 0.5955 - tp0.9: 771789.0000 - fp0.9: 70077.0000 - tn0.9: 16571767.0000 - fn0.9: 2198015.0000 - precision0.9: 0.9168 - recall0.9: 0.2599 - accuracy: 0.9222 - auc: 0.
    9375 - f1: 0.2630 - val_loss: 0.5272 - val_tp0.1: 756550.0000 - val_fp0.1: 886258.0000 - val_tn0.1: 3189329.0000 - val_fn0.1: 42103.0000 - val_precision0.1: 0.4605 - val_recall0.1: 0.9473
    - val_tp0.3: 683180.0000 - val_fp0.3: 415168.0000 - val_tn0.3: 3660419.0000 - val_fn0.3: 115473.0000 - val_precision0.3: 0.6220 - val_recall0.3: 0.8554 - val_tp0.5: 618115.0000 - val_fp0.5
    : 244352.0000 - val_tn0.5: 3831235.0000 - val_fn0.5: 180538.0000 - val_precision0.5: 0.7167 - val_recall0.5: 0.7739 - val_tp0.7: 498018.0000 - val_fp0.7: 115297.0000 - val_tn0.7: 3960290.0
    000 - val_fn0.7: 300635.0000 - val_precision0.7: 0.8120 - val_recall0.7: 0.6236 - val_tp0.9: 233878.0000 - val_fp0.9: 19740.0000 - val_tn0.9: 4055847.0000 - val_fn0.9: 564775.0000 - val_pr
    ecision0.9: 0.9222 - val_recall0.9: 0.2928 - val_accuracy: 0.9128 - val_auc: 0.9378 - val_f1: 0.2816
    Epoch 16/20
    342/342 [==============================] - 29s 86ms/step - loss: 0.4848 - tp0.1: 2781482.0000 - fp0.1: 2957469.0000 - tn0.1: 13686899.0000 - fn0.1: 185798.0000 - precision0.1: 0.4847 - rec
    all0.1: 0.9374 - tp0.3: 2494189.0000 - fp0.3: 1353502.0000 - tn0.3: 15290866.0000 - fn0.3: 473091.0000 - precision0.3: 0.6482 - recall0.3: 0.8406 - tp0.5: 2230152.0000 - fp0.5: 792220.0000
     - tn0.5: 15852148.0000 - fn0.5: 737128.0000 - precision0.5: 0.7379 - recall0.5: 0.7516 - tp0.7: 1783255.0000 - fp0.7: 363318.0000 - tn0.7: 16281050.0000 - fn0.7: 1184025.0000 - precision0
    .7: 0.8307 - recall0.7: 0.6010 - tp0.9: 789040.0000 - fp0.9: 65516.0000 - tn0.9: 16578852.0000 - fn0.9: 2178240.0000 - precision0.9: 0.9233 - recall0.9: 0.2659 - accuracy: 0.9220 - auc: 0.
    9387 - f1: 0.2628 - val_loss: 0.5289 - val_tp0.1: 757278.0000 - val_fp0.1: 905312.0000 - val_tn0.1: 3170851.0000 - val_fn0.1: 40799.0000 - val_precision0.1: 0.4555 - val_recall0.1: 0.9489
    - val_tp0.3: 686534.0000 - val_fp0.3: 428363.0000 - val_tn0.3: 3647800.0000 - val_fn0.3: 111543.0000 - val_precision0.3: 0.6158 - val_recall0.3: 0.8602 - val_tp0.5: 623130.0000 - val_fp0.5
    : 254259.0000 - val_tn0.5: 3821904.0000 - val_fn0.5: 174947.0000 - val_precision0.5: 0.7102 - val_recall0.5: 0.7808 - val_tp0.7: 505552.0000 - val_fp0.7: 121120.0000 - val_tn0.7: 3955043.0
    000 - val_fn0.7: 292525.0000 - val_precision0.7: 0.8067 - val_recall0.7: 0.6335 - val_tp0.9: 244234.0000 - val_fp0.9: 21695.0000 - val_tn0.9: 4054468.0000 - val_fn0.9: 553843.0000 - val_pr
    ecision0.9: 0.9184 - val_recall0.9: 0.3060 - val_accuracy: 0.9119 - val_auc: 0.9382 - val_f1: 0.2814
    Epoch 17/20
    342/342 [==============================] - 30s 86ms/step - loss: 0.4849 - tp0.1: 2790598.0000 - fp0.1: 2957998.0000 - tn0.1: 13679248.0000 - fn0.1: 183804.0000 - precision0.1: 0.4854 - rec
    all0.1: 0.9382 - tp0.3: 2505785.0000 - fp0.3: 1360823.0000 - tn0.3: 15276423.0000 - fn0.3: 468617.0000 - precision0.3: 0.6481 - recall0.3: 0.8425 - tp0.5: 2243181.0000 - fp0.5: 797599.0000
     - tn0.5: 15839647.0000 - fn0.5: 731221.0000 - precision0.5: 0.7377 - recall0.5: 0.7542 - tp0.7: 1792202.0000 - fp0.7: 377983.0000 - tn0.7: 16259263.0000 - fn0.7: 1182200.0000 - precision0
    .7: 0.8258 - recall0.7: 0.6025 - tp0.9: 784914.0000 - fp0.9: 75124.0000 - tn0.9: 16562122.0000 - fn0.9: 2189488.0000 - precision0.9: 0.9127 - recall0.9: 0.2639 - accuracy: 0.9220 - auc: 0.
    9387 - f1: 0.2634 - val_loss: 0.5301 - val_tp0.1: 751131.0000 - val_fp0.1: 924609.0000 - val_tn0.1: 3159383.0000 - val_fn0.1: 39117.0000 - val_precision0.1: 0.4482 - val_recall0.1: 0.9505
    - val_tp0.3: 682847.0000 - val_fp0.3: 443618.0000 - val_tn0.3: 3640374.0000 - val_fn0.3: 107401.0000 - val_precision0.3: 0.6062 - val_recall0.3: 0.8641 - val_tp0.5: 620126.0000 - val_fp0.5
    : 265041.0000 - val_tn0.5: 3818951.0000 - val_fn0.5: 170122.0000 - val_precision0.5: 0.7006 - val_recall0.5: 0.7847 - val_tp0.7: 505243.0000 - val_fp0.7: 127349.0000 - val_tn0.7: 3956643.0
    000 - val_fn0.7: 285005.0000 - val_precision0.7: 0.7987 - val_recall0.7: 0.6393 - val_tp0.9: 248302.0000 - val_fp0.9: 23740.0000 - val_tn0.9: 4060252.0000 - val_fn0.9: 541946.0000 - val_pr
    ecision0.9: 0.9127 - val_recall0.9: 0.3142 - val_accuracy: 0.9107 - val_auc: 0.9382 - val_f1: 0.2790
    Epoch 18/20
    342/342 [==============================] - 30s 88ms/step - loss: 0.4853 - tp0.1: 2805900.0000 - fp0.1: 2941444.0000 - tn0.1: 13677897.0000 - fn0.1: 186407.0000 - precision0.1: 0.4882 - rec
    all0.1: 0.9377 - tp0.3: 2517150.0000 - fp0.3: 1335602.0000 - tn0.3: 15283739.0000 - fn0.3: 475157.0000 - precision0.3: 0.6533 - recall0.3: 0.8412 - tp0.5: 2248387.0000 - fp0.5: 781307.0000
     - tn0.5: 15838034.0000 - fn0.5: 743920.0000 - precision0.5: 0.7421 - recall0.5: 0.7514 - tp0.7: 1790819.0000 - fp0.7: 356494.0000 - tn0.7: 16262847.0000 - fn0.7: 1201488.0000 - precision0
    .7: 0.8340 - recall0.7: 0.5985 - tp0.9: 783564.0000 - fp0.9: 70623.0000 - tn0.9: 16548718.0000 - fn0.9: 2208743.0000 - precision0.9: 0.9173 - recall0.9: 0.2619 - accuracy: 0.9222 - auc: 0.
    9390 - f1: 0.2648 - val_loss: 0.5309 - val_tp0.1: 759007.0000 - val_fp0.1: 906199.0000 - val_tn0.1: 3168039.0000 - val_fn0.1: 40995.0000 - val_precision0.1: 0.4558 - val_recall0.1: 0.9488
    - val_tp0.3: 688612.0000 - val_fp0.3: 432765.0000 - val_tn0.3: 3641473.0000 - val_fn0.3: 111390.0000 - val_precision0.3: 0.6141 - val_recall0.3: 0.8608 - val_tp0.5: 625381.0000 - val_fp0.5
    : 258408.0000 - val_tn0.5: 3815830.0000 - val_fn0.5: 174621.0000 - val_precision0.5: 0.7076 - val_recall0.5: 0.7817 - val_tp0.7: 508544.0000 - val_fp0.7: 124098.0000 - val_tn0.7: 3950140.0
    000 - val_fn0.7: 291458.0000 - val_precision0.7: 0.8038 - val_recall0.7: 0.6357 - val_tp0.9: 247680.0000 - val_fp0.9: 22653.0000 - val_tn0.9: 4051585.0000 - val_fn0.9: 552322.0000 - val_pr
    ecision0.9: 0.9162 - val_recall0.9: 0.3096 - val_accuracy: 0.9112 - val_auc: 0.9378 - val_f1: 0.2820
    Epoch 19/20
    342/342 [==============================] - 30s 87ms/step - loss: 0.4853 - tp0.1: 2765154.0000 - fp0.1: 2965014.0000 - tn0.1: 13695648.0000 - fn0.1: 185832.0000 - precision0.1: 0.4826 - rec
    all0.1: 0.9370 - tp0.3: 2489223.0000 - fp0.3: 1363429.0000 - tn0.3: 15297233.0000 - fn0.3: 461763.0000 - precision0.3: 0.6461 - recall0.3: 0.8435 - tp0.5: 2236269.0000 - fp0.5: 805108.0000
     - tn0.5: 15855554.0000 - fn0.5: 714717.0000 - precision0.5: 0.7353 - recall0.5: 0.7578 - tp0.7: 1791573.0000 - fp0.7: 372427.0000 - tn0.7: 16288235.0000 - fn0.7: 1159413.0000 - precision0
    .7: 0.8279 - recall0.7: 0.6071 - tp0.9: 777165.0000 - fp0.9: 72719.0000 - tn0.9: 16587943.0000 - fn0.9: 2173821.0000 - precision0.9: 0.9144 - recall0.9: 0.2634 - accuracy: 0.9225 - auc: 0.
    9385 - f1: 0.2616 - val_loss: 0.5302 - val_tp0.1: 757018.0000 - val_fp0.1: 914987.0000 - val_tn0.1: 3162308.0000 - val_fn0.1: 39927.0000 - val_precision0.1: 0.4528 - val_recall0.1: 0.9499
    - val_tp0.3: 687887.0000 - val_fp0.3: 439024.0000 - val_tn0.3: 3638271.0000 - val_fn0.3: 109058.0000 - val_precision0.3: 0.6104 - val_recall0.3: 0.8632 - val_tp0.5: 624960.0000 - val_fp0.5
    : 262546.0000 - val_tn0.5: 3814749.0000 - val_fn0.5: 171985.0000 - val_precision0.5: 0.7042 - val_recall0.5: 0.7842 - val_tp0.7: 509687.0000 - val_fp0.7: 126771.0000 - val_tn0.7: 3950524.0
    000 - val_fn0.7: 287258.0000 - val_precision0.7: 0.8008 - val_recall0.7: 0.6396 - val_tp0.9: 251547.0000 - val_fp0.9: 23931.0000 - val_tn0.9: 4053364.0000 - val_fn0.9: 545398.0000 - val_pr
    ecision0.9: 0.9131 - val_recall0.9: 0.3156 - val_accuracy: 0.9109 - val_auc: 0.9382 - val_f1: 0.2811
    Epoch 20/20
    342/342 [==============================] - 29s 84ms/step - loss: 0.4987 - tp0.1: 2765147.0000 - fp0.1: 2972400.0000 - tn0.1: 13672870.0000 - fn0.1: 201231.0000 - precision0.1: 0.4819 - rec
    all0.1: 0.9322 - tp0.3: 2475699.0000 - fp0.3: 1368612.0000 - tn0.3: 15276658.0000 - fn0.3: 490679.0000 - precision0.3: 0.6440 - recall0.3: 0.8346 - tp0.5: 2217066.0000 - fp0.5: 801284.0000
     - tn0.5: 15843986.0000 - fn0.5: 749312.0000 - precision0.5: 0.7345 - recall0.5: 0.7474 - tp0.7: 1769267.0000 - fp0.7: 376020.0000 - tn0.7: 16269250.0000 - fn0.7: 1197111.0000 - precision0
    .7: 0.8247 - recall0.7: 0.5964 - tp0.9: 778944.0000 - fp0.9: 72985.0000 - tn0.9: 16572285.0000 - fn0.9: 2187434.0000 - precision0.9: 0.9143 - recall0.9: 0.2626 - accuracy: 0.9209 - auc: 0.
    9352 - f1: 0.2628 - val_loss: 0.5304 - val_tp0.1: 756740.0000 - val_fp0.1: 911039.0000 - val_tn0.1: 3166007.0000 - val_fn0.1: 40454.0000 - val_precision0.1: 0.4537 - val_recall0.1: 0.9493
    - val_tp0.3: 686440.0000 - val_fp0.3: 434245.0000 - val_tn0.3: 3642801.0000 - val_fn0.3: 110754.0000 - val_precision0.3: 0.6125 - val_recall0.3: 0.8611 - val_tp0.5: 622523.0000 - val_fp0.5
    : 258158.0000 - val_tn0.5: 3818888.0000 - val_fn0.5: 174671.0000 - val_precision0.5: 0.7069 - val_recall0.5: 0.7809 - val_tp0.7: 505054.0000 - val_fp0.7: 122534.0000 - val_tn0.7: 3954512.0
    000 - val_fn0.7: 292140.0000 - val_precision0.7: 0.8048 - val_recall0.7: 0.6335 - val_tp0.9: 241926.0000 - val_fp0.9: 21836.0000 - val_tn0.9: 4055210.0000 - val_fn0.9: 555268.0000 - val_pr
    ecision0.9: 0.9172 - val_recall0.9: 0.3035 - val_accuracy: 0.9112 - val_auc: 0.9379 - val_f1: 0.2811
    --- Running training session 64/140
    {'hp_epochs': 20, 'hp_batch_size': 14, 'hp_scaler': 'quant_g', 'hp_n_levels': 7, 'hp_first_filters': 16, 'hp_pool_size': 2, 'hp_input_size': 4096, 'hp_lr_start': 0.027144629354532844, 'hp_
    lr_power': 5.0}
    --- repeat #: 2
    input - shape:   (None, 4096, 1)
    output - shape:  (None, 4096, 1)
    bash: line 1: 36008 Segmentation fault      python src/fluotracify/training/search_hparams.py --num_session_groups 70 --fluotracify_path /beegfs/ye53nis/drmed-git/src --csv_path_train /bee
    gfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets --csv_path_val /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN --col_per_example 3
    2021/08/08 13:58:32 ERROR mlflow.cli: === Run (ID '0a91e997b0884a00bd6c932104f84722') failed ===
    (tf) [ye53nis@node130 drmed-git]$
    

2.4.3 Run 2 - new hparams

2.4.3.1 Record metadata
  1. Current directory, last 5 git commits
    pwd
    git log -5
    
    (tf) [ye53nis@node130 drmed-git]$ pwd
    /beegfs/ye53nis/drmed-git
    (tf) [ye53nis@node130 drmed-git]$ git log -5
    commit 150ad647300f3306635fa7f5d75fa0d66165df25
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Sun Aug 8 17:28:49 2021 +0200
    
        Fix missing brackets
    
    commit 6cef8d3f9166fe6cb19b6ec064573907a0ca50a6
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Sun Aug 8 17:26:06 2021 +0200
    
        Adjust batch size hparam to upper limit 30
    
    commit 609a75bccfd08fa36b2d55861ec5d7954e1adbe7
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Sun Aug 8 17:05:12 2021 +0200
    
        Adjust hyperparameters
    
    commit 739b18bf64ef886d7ae0468b4fc3d5b9b4ccf12c
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Sun Aug 8 16:42:16 2021 +0200
    
        Change random seed to enable different 2nd run
    
    commit aa5b9bc35c53c4fd1525c6b812b2a28532ae7afb
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Sat Aug 7 22:15:02 2021 +0200
    
        Add hparams combi restriction; add metadata
    
        problems arise, if in the random combination of hparams, 2*pool_size**n_levels
        is bigger than the input_size. That's why these cases are skipped now.
    (tf) [ye53nis@node130 drmed-git]$
    
  2. system and conda env didn't change, see above.
2.4.3.2 Mlflow run 2 (failed mid run)
  • The last parent run failed to an unkown error (bash: line 1: 36008 Segmentation fault). That means only 63/140 sessions could be run, while also only 45 sessions were eligible (for the rest, the hparams were not compatible with each other). Still, these first 45 runs delivered a lot of interesting information, so I decided to skim through MLflow to look for improvements in the hparam space and I came up with the following:
    • leave out smallest batch sizes and go from 4 to 30
    • make number of levels an integer interval from 1 to 9
    • make first filters an integer interval from 1 to 128
    • add poolsize (+ stride + kernel size) 8
    • adjust starting learning rate real interval to 1e-6 to 0.06
    • make learning rate power integer interval from 1 to 7
  • now the run:
    mlflow run . -e search_hparams -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets -P csv_path_val=/beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN -P num_session_groups=60
    
    2021-08-09 03:52:38.082134: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba5fe00 of size 256 by op assert_greater_equal_30/Assert/AssertGuard/pivot_f/_843 action_count 94195455584406 step 0 next 151
    2021-08-09 03:52:38.082152: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba5ff00 of size 256 by op assert_greater_equal_30/Assert/AssertGuard/pivot_t/_844 action_count 94195455584407 step 0 next 2585
    2021-08-09 03:52:38.082174: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba60000 of size 256 by op assert_less_equal_30/Assert/AssertGuard/pivot_f/_853 action_count 94195455584408 step 0 next 2261
    2021-08-09 03:52:38.082198: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba60100 of size 256 by op assert_less_equal_30/Assert/AssertGuard/pivot_t/_854 action_count 94195455584409 step 0 next 1893
    2021-08-09 03:52:38.082216: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba60200 of size 256 by op assert_greater_equal_31/Assert/AssertGuard/pivot_f/_871 action_count 94195455584410 step 0 next 2158
    2021-08-09 03:52:38.082234: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba60300 of size 256 by op assert_greater_equal_31/Assert/AssertGuard/pivot_t/_872 action_count 94195455584411 step 0 next 2224
    2021-08-09 03:52:38.082258: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba60400 of size 256 by op assert_less_equal_31/Assert/AssertGuard/pivot_f/_881 action_count 94195455584412 step 0 next 172
    2021-08-09 03:52:38.082277: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba60500 of size 256 by op assert_less_equal_31/Assert/AssertGuard/pivot_t/_882 action_count 94195455584413 step 0 next 2654
    2021-08-09 03:52:38.082295: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba60600 of size 256 by op assert_greater_equal_32/Assert/AssertGuard/pivot_f/_899 action_count 94195455584414 step 0 next 1354
    2021-08-09 03:52:38.082316: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba60700 of size 256 by op assert_greater_equal_32/Assert/AssertGuard/pivot_t/_900 action_count 94195455584415 step 0 next 2250
    2021-08-09 03:52:38.082338: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba60800 of size 256 by op assert_less_equal_32/Assert/AssertGuard/pivot_f/_909 action_count 94195455584416 step 0 next 2458
    2021-08-09 03:52:38.082359: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba60900 of size 256 by op assert_less_equal_32/Assert/AssertGuard/pivot_t/_910 action_count 94195455584417 step 0 next 1032
    2021-08-09 03:52:38.082380: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba60a00 of size 256 by op Sub action_count 94195480110816 step 0 next 970
    2021-08-09 03:52:38.082401: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba60b00 of size 256 by op AssignAddVariableOp_2 action_count 94195467561410 step 17436438550591945738 next 1371
    2021-08-09 03:52:38.082422: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba60c00 of size 512 by op Fill action_count 94195467562918 step 0 next 2415
    2021-08-09 03:52:38.082443: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba60e00 of size 512 by op Fill action_count 94195467562919 step 0 next 2029
    2021-08-09 03:52:38.082466: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba61000 of size 256 by op AssignAddVariableOp_52 action_count 94195467505890 step 11647544380785529683 next 2491
    2021-08-09 03:52:38.082487: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba61100 of size 768 by op Fill action_count 94195467562905 step 0 next 1453
    2021-08-09 03:52:38.082508: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba61400 of size 256 by op AssignAddVariableOp_20 action_count 94195467561512 step 17436438550591945738 next 1396
    2021-08-09 03:52:38.082538: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba61500 of size 256 by op AssignAddVariableOp_18 action_count 94195467561452 step 17436438550591945738 next 2965
    2021-08-09 03:52:38.082558: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba61600 of size 256 by op Sub action_count 94195480110817 step 0 next 1920
    2021-08-09 03:52:38.082575: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba61700 of size 256 by op AssignAddVariableOp_28 action_count 94195467561516 step 17436438550591945738 next 730
    2021-08-09 03:52:38.082593: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba61800 of size 256 by op AssignAddVariableOp_26 action_count 94195467561464 step 17436438550591945738 next 3037
    2021-08-09 03:52:38.082616: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba61900 of size 256 by op AssignAddVariableOp_36 action_count 94195467561520 step 17436438550591945738 next 1876
    2021-08-09 03:52:38.082636: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba61a00 of size 256 by op AssignAddVariableOp_34 action_count 94195467561476 step 17436438550591945738 next 1958
    2021-08-09 03:52:38.082657: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba61b00 of size 1536 by op Fill action_count 94195467562929 step 0 next 1889
    2021-08-09 03:52:38.082678: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba62100 of size 256 by op Sub action_count 94195299053113 step 0 next 1087
    2021-08-09 03:52:38.082698: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba62200 of size 256 by op Sub action_count 94195299053114 step 0 next 680
    2021-08-09 03:52:38.082718: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba62300 of size 2048 by op Fill action_count 94195455013702 step 0 next 2691
    2021-08-09 03:52:38.082738: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba62b00 of size 2048 by op Fill action_count 94195455013703 step 0 next 3049
    2021-08-09 03:52:38.082759: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba63300 of size 2048 by op Fill action_count 94195455013704 step 0 next 2707
    2021-08-09 03:52:38.082780: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba63b00 of size 2048 by op Fill action_count 94195455013706 step 0 next 1505
    2021-08-09 03:52:38.082801: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba64300 of size 256 by op Sub action_count 94195365012693 step 0 next 2311
    2021-08-09 03:52:38.082821: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba64400 of size 256 by op Sub action_count 94195365012694 step 0 next 638
    2021-08-09 03:52:38.082843: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba64500 of size 256 by op assert_greater_equal_28/Assert/AssertGuard/pivot_f/_787 action_count 94195455013959 step 0 next 2355
    2021-08-09 03:52:38.082866: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba64600 of size 256 by op assert_greater_equal_28/Assert/AssertGuard/pivot_t/_788 action_count 94195455013960 step 0 next 2332
    2021-08-09 03:52:38.082890: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba64700 of size 256 by op assert_less_equal_28/Assert/AssertGuard/pivot_f/_797 action_count 94195455013961 step 0 next 2330
    2021-08-09 03:52:38.082910: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba64800 of size 256 by op assert_less_equal_28/Assert/AssertGuard/pivot_t/_798 action_count 94195455013962 step 0 next 2402
    2021-08-09 03:52:38.082933: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba64900 of size 256 by op gradient_tape/unet_depth3/encode0/batch_normalization_546/moments/scalar action_count 94195455013963 step 0 nex
    t 114
    2021-08-09 03:52:38.082952: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba64a00 of size 256 by op assert_greater_equal_29/Assert/AssertGuard/pivot_f/_815 action_count 94195455013964 step 0 next 2297
    2021-08-09 03:52:38.082974: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba64b00 of size 256 by op assert_greater_equal_29/Assert/AssertGuard/pivot_t/_816 action_count 94195455013965 step 0 next 2106
    2021-08-09 03:52:38.082997: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba64c00 of size 256 by op Adam/Adam/Const action_count 94195455013966 step 0 next 2308
    2021-08-09 03:52:38.083015: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba64d00 of size 256 by op assert_less_equal_29/Assert/AssertGuard/pivot_f/_825 action_count 94195455013967 step 0 next 2318
    2021-08-09 03:52:38.083036: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba64e00 of size 256 by op assert_less_equal_29/Assert/AssertGuard/pivot_t/_826 action_count 94195455013968 step 0 next 2323
    2021-08-09 03:52:38.083059: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba64f00 of size 256 by op assert_greater_equal_30/Assert/AssertGuard/pivot_f/_843 action_count 94195455013969 step 0 next 2391
    2021-08-09 03:52:38.083079: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65000 of size 256 by op assert_greater_equal_30/Assert/AssertGuard/pivot_t/_844 action_count 94195455013970 step 0 next 1074
    2021-08-09 03:52:38.083097: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65100 of size 256 by op assert_less_equal_30/Assert/AssertGuard/pivot_f/_853 action_count 94195455013971 step 0 next 2382
    2021-08-09 03:52:38.083118: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65200 of size 256 by op assert_less_equal_30/Assert/AssertGuard/pivot_t/_854 action_count 94195455013972 step 0 next 2272
    2021-08-09 03:52:38.083140: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65300 of size 256 by op assert_greater_equal_31/Assert/AssertGuard/pivot_f/_871 action_count 94195455013973 step 0 next 1355
    2021-08-09 03:52:38.083158: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65400 of size 256 by op assert_greater_equal_31/Assert/AssertGuard/pivot_t/_872 action_count 94195455013974 step 0 next 2378
    2021-08-09 03:52:38.083178: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65500 of size 256 by op assert_less_equal_31/Assert/AssertGuard/pivot_f/_881 action_count 94195455013975 step 0 next 2306
    2021-08-09 03:52:38.083201: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65600 of size 256 by op assert_less_equal_31/Assert/AssertGuard/pivot_t/_882 action_count 94195455013976 step 0 next 2294
    2021-08-09 03:52:38.083220: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65700 of size 256 by op assert_greater_equal_32/Assert/AssertGuard/pivot_f/_899 action_count 94195455013977 step 0 next 2331
    2021-08-09 03:52:38.083243: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65800 of size 256 by op assert_greater_equal_32/Assert/AssertGuard/pivot_t/_900 action_count 94195455013978 step 0 next 2322
    2021-08-09 03:52:38.083263: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65900 of size 256 by op assert_less_equal_32/Assert/AssertGuard/pivot_f/_909 action_count 94195455013979 step 0 next 2221
    2021-08-09 03:52:38.083283: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65a00 of size 256 by op assert_less_equal_32/Assert/AssertGuard/pivot_t/_910 action_count 94195455013980 step 0 next 2336
    2021-08-09 03:52:38.083306: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65b00 of size 256 by op assert_greater_equal_15/Assert/AssertGuard/pivot_f/_423 action_count 94195455584346 step 0 next 2019
    2021-08-09 03:52:38.083324: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65c00 of size 256 by op assert_greater_equal_15/Assert/AssertGuard/pivot_t/_424 action_count 94195455584347 step 0 next 2376
    2021-08-09 03:52:38.083346: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65d00 of size 256 by op assert_less_equal_15/Assert/AssertGuard/pivot_f/_433 action_count 94195455584348 step 0 next 1688
    2021-08-09 03:52:38.083369: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65e00 of size 256 by op assert_less_equal_15/Assert/AssertGuard/pivot_t/_434 action_count 94195455584349 step 0 next 2620
    2021-08-09 03:52:38.083391: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba65f00 of size 256 by op assert_greater_equal_16/Assert/AssertGuard/pivot_f/_451 action_count 94195455584350 step 0 next 391
    2021-08-09 03:52:38.083414: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66000 of size 256 by op assert_greater_equal_16/Assert/AssertGuard/pivot_t/_452 action_count 94195455584351 step 0 next 1462
    2021-08-09 03:52:38.083435: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66100 of size 256 by op assert_less_equal_16/Assert/AssertGuard/pivot_f/_461 action_count 94195455584352 step 0 next 985
    2021-08-09 03:52:38.083453: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66200 of size 256 by op assert_less_equal_16/Assert/AssertGuard/pivot_t/_462 action_count 94195455584353 step 0 next 2287
    2021-08-09 03:52:38.083475: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66300 of size 256 by op assert_greater_equal_17/Assert/AssertGuard/pivot_f/_479 action_count 94195455584354 step 0 next 2292
    2021-08-09 03:52:38.083495: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66400 of size 256 by op assert_greater_equal_17/Assert/AssertGuard/pivot_t/_480 action_count 94195455584355 step 0 next 2350
    2021-08-09 03:52:38.083522: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66500 of size 256 by op assert_less_equal_17/Assert/AssertGuard/pivot_f/_489 action_count 94195455584356 step 0 next 2256
    2021-08-09 03:52:38.083546: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66600 of size 256 by op assert_less_equal_17/Assert/AssertGuard/pivot_t/_490 action_count 94195455584357 step 0 next 2310
    2021-08-09 03:52:38.083566: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66700 of size 256 by op assert_greater_equal_18/Assert/AssertGuard/pivot_f/_507 action_count 94195455584358 step 0 next 60
    2021-08-09 03:52:38.083587: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66800 of size 256 by op assert_greater_equal_18/Assert/AssertGuard/pivot_t/_508 action_count 94195455584359 step 0 next 1380
    2021-08-09 03:52:38.083608: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66900 of size 256 by op assert_less_equal_18/Assert/AssertGuard/pivot_f/_517 action_count 94195455584360 step 0 next 2375
    2021-08-09 03:52:38.083628: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66a00 of size 256 by op assert_less_equal_18/Assert/AssertGuard/pivot_t/_518 action_count 94195455584361 step 0 next 2497
    2021-08-09 03:52:38.083646: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66b00 of size 256 by op assert_greater_equal_19/Assert/AssertGuard/pivot_f/_535 action_count 94195455584362 step 0 next 677
    2021-08-09 03:52:38.083663: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66c00 of size 256 by op assert_greater_equal_19/Assert/AssertGuard/pivot_t/_536 action_count 94195455584363 step 0 next 2326
    2021-08-09 03:52:38.083687: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66d00 of size 256 by op assert_less_equal_19/Assert/AssertGuard/pivot_f/_545 action_count 94195455584364 step 0 next 2928
    2021-08-09 03:52:38.083707: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66e00 of size 256 by op assert_less_equal_19/Assert/AssertGuard/pivot_t/_546 action_count 94195455584365 step 0 next 2116
    2021-08-09 03:52:38.083728: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba66f00 of size 256 by op assert_greater_equal_20/Assert/AssertGuard/pivot_f/_563 action_count 94195455584366 step 0 next 411
    2021-08-09 03:52:38.083751: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67000 of size 256 by op assert_greater_equal_20/Assert/AssertGuard/pivot_t/_564 action_count 94195455584367 step 0 next 1778
    2021-08-09 03:52:38.083770: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67100 of size 256 by op assert_less_equal_20/Assert/AssertGuard/pivot_f/_573 action_count 94195455584368 step 0 next 2345
    2021-08-09 03:52:38.083791: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67200 of size 256 by op assert_less_equal_20/Assert/AssertGuard/pivot_t/_574 action_count 94195455584369 step 0 next 315
    2021-08-09 03:52:38.083810: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67300 of size 256 by op assert_greater_equal_21/Assert/AssertGuard/pivot_f/_591 action_count 94195455584370 step 0 next 373
    2021-08-09 03:52:38.083831: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67400 of size 256 by op assert_greater_equal_21/Assert/AssertGuard/pivot_t/_592 action_count 94195455584371 step 0 next 1535
    2021-08-09 03:52:38.083854: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67500 of size 256 by op assert_less_equal_21/Assert/AssertGuard/pivot_f/_601 action_count 94195455584372 step 0 next 1914
    2021-08-09 03:52:38.083872: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67600 of size 256 by op assert_less_equal_21/Assert/AssertGuard/pivot_t/_602 action_count 94195455584373 step 0 next 2521
    2021-08-09 03:52:38.083894: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67700 of size 256 by op assert_greater_equal_22/Assert/AssertGuard/pivot_f/_619 action_count 94195455584374 step 0 next 2110
    2021-08-09 03:52:38.083915: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67800 of size 256 by op assert_greater_equal_22/Assert/AssertGuard/pivot_t/_620 action_count 94195455584375 step 0 next 968
    2021-08-09 03:52:38.083932: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67900 of size 256 by op assert_less_equal_22/Assert/AssertGuard/pivot_f/_629 action_count 94195455584376 step 0 next 2437
    2021-08-09 03:52:38.083954: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67a00 of size 256 by op assert_less_equal_22/Assert/AssertGuard/pivot_t/_630 action_count 94195455584377 step 0 next 218
    2021-08-09 03:52:38.083975: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67b00 of size 256 by op assert_greater_equal_23/Assert/AssertGuard/pivot_f/_647 action_count 94195455584378 step 0 next 1581
    2021-08-09 03:52:38.083992: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67c00 of size 256 by op assert_greater_equal_23/Assert/AssertGuard/pivot_t/_648 action_count 94195455584379 step 0 next 2337
    2021-08-09 03:52:38.084013: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67d00 of size 256 by op Sub action_count 94195299053151 step 0 next 976
    2021-08-09 03:52:38.084034: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67e00 of size 256 by op Sub action_count 94195299053152 step 0 next 1112
    2021-08-09 03:52:38.084055: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba67f00 of size 2048 by op Fill action_count 94195455013774 step 0 next 1401
    2021-08-09 03:52:38.084076: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba68700 of size 2048 by op Fill action_count 94195455013776 step 0 next 1101
    2021-08-09 03:52:38.084097: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba68f00 of size 2048 by op Fill action_count 94195455013777 step 0 next 519
    2021-08-09 03:52:38.084114: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba69700 of size 2048 by op Fill action_count 94195455013778 step 0 next 799
    2021-08-09 03:52:38.084135: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba69f00 of size 3072 by op Fill action_count 94195455013780 step 0 next 607
    2021-08-09 03:52:38.084156: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba6ab00 of size 256 by op Sub action_count 94195299053177 step 0 next 2325
    2021-08-09 03:52:38.084173: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba6ac00 of size 256 by op Sub action_count 94195299053178 step 0 next 1838
    2021-08-09 03:52:38.084194: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abba6ad00 of size 995328 by op Fill action_count 94195455013725 step 0 next 335
    2021-08-09 03:52:38.084211: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbb5dd00 of size 256 by op Sub action_count 94195129464776 step 0 next 1993
    2021-08-09 03:52:38.084230: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbb5de00 of size 4352 by op Fill action_count 94195455013478 step 0 next 1049
    2021-08-09 03:52:38.084251: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbb5ef00 of size 256 by op Sub action_count 94195299052776 step 0 next 2327
    2021-08-09 03:52:38.084268: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbb5f000 of size 512 by op Fill action_count 94195455013382 step 0 next 444
    2021-08-09 03:52:38.084291: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbb5f200 of size 256 by op AssignAddVariableOp_12 action_count 94195455011283 step 18076379956407770847 next 3070
    2021-08-09 03:52:38.084333: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbb5f300 of size 256 by op AssignAddVariableOp_10 action_count 94195455011215 step 18076379956407770847 next 1540
    2021-08-09 03:52:38.084353: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbb5f400 of size 256 by op Sub action_count 94195299052789 step 0 next 1949
    2021-08-09 03:52:38.084374: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbb5f500 of size 256 by op Sub action_count 94195129464761 step 0 next 988
    2021-08-09 03:52:38.084394: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbb5f600 of size 256 by op Sub action_count 94195152301775 step 0 next 1907
    2021-08-09 03:52:38.084414: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbb5f700 of size 256 by op Sub action_count 94195152301788 step 0 next 1591
    2021-08-09 03:52:38.084430: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbb5f800 of size 256 by op Sub action_count 94195129464762 step 0 next 2090
    2021-08-09 03:52:38.084448: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbb5f900 of size 4673536 by op Fill action_count 94195455013689 step 0 next 59
    2021-08-09 03:52:38.084468: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd4900 of size 256 by op Sub action_count 94195065726202 step 0 next 659
    2021-08-09 03:52:38.084490: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd4a00 of size 256 by op assert_less_equal_16/Assert/AssertGuard/pivot_f/_461 action_count 94195455013907 step 0 next 2217
    2021-08-09 03:52:38.084508: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd4b00 of size 256 by op assert_less_equal_16/Assert/AssertGuard/pivot_t/_462 action_count 94195455013908 step 0 next 1616
    2021-08-09 03:52:38.084539: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd4c00 of size 256 by op assert_greater_equal_17/Assert/AssertGuard/pivot_f/_479 action_count 94195455013909 step 0 next 2204
    2021-08-09 03:52:38.084577: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd4d00 of size 256 by op assert_greater_equal_17/Assert/AssertGuard/pivot_t/_480 action_count 94195455013910 step 0 next 339
    2021-08-09 03:52:38.084611: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd4e00 of size 256 by op assert_less_equal_17/Assert/AssertGuard/pivot_f/_489 action_count 94195455013911 step 0 next 701
    2021-08-09 03:52:38.084635: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd4f00 of size 256 by op assert_less_equal_17/Assert/AssertGuard/pivot_t/_490 action_count 94195455013912 step 0 next 1708
    2021-08-09 03:52:38.084657: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5000 of size 256 by op gradient_tape/unet_depth3/two_conv_center/batch_normalization_552/moments/scalar action_count 94195455013913 st
    ep 0 next 903
    2021-08-09 03:52:38.084675: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5100 of size 256 by op assert_greater_equal_18/Assert/AssertGuard/pivot_f/_507 action_count 94195455013914 step 0 next 2371
    2021-08-09 03:52:38.084698: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5200 of size 256 by op assert_greater_equal_18/Assert/AssertGuard/pivot_t/_508 action_count 94195455013915 step 0 next 263
    2021-08-09 03:52:38.084721: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5300 of size 256 by op assert_less_equal_18/Assert/AssertGuard/pivot_f/_517 action_count 94195455013916 step 0 next 154
    2021-08-09 03:52:38.084744: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5400 of size 256 by op assert_less_equal_18/Assert/AssertGuard/pivot_t/_518 action_count 94195455013917 step 0 next 2321
    2021-08-09 03:52:38.084765: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5500 of size 256 by op assert_greater_equal_19/Assert/AssertGuard/pivot_f/_535 action_count 94195455013918 step 0 next 2253
    2021-08-09 03:52:38.084783: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5600 of size 256 by op assert_greater_equal_19/Assert/AssertGuard/pivot_t/_536 action_count 94195455013919 step 0 next 2236
    2021-08-09 03:52:38.084802: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5700 of size 256 by op assert_less_equal_19/Assert/AssertGuard/pivot_f/_545 action_count 94195455013920 step 0 next 2222
    2021-08-09 03:52:38.084823: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5800 of size 256 by op assert_less_equal_19/Assert/AssertGuard/pivot_t/_546 action_count 94195455013921 step 0 next 1262
    2021-08-09 03:52:38.084847: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5900 of size 256 by op gradient_tape/unet_depth3/encode2/batch_normalization_551/moments/scalar action_count 94195455013922 step 0 nex
    t 2443
    2021-08-09 03:52:38.084868: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5a00 of size 256 by op assert_greater_equal_20/Assert/AssertGuard/pivot_f/_563 action_count 94195455013923 step 0 next 2313
    2021-08-09 03:52:38.084889: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5b00 of size 256 by op assert_greater_equal_20/Assert/AssertGuard/pivot_t/_564 action_count 94195455013924 step 0 next 2293
    2021-08-09 03:52:38.084912: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5c00 of size 256 by op assert_less_equal_20/Assert/AssertGuard/pivot_f/_573 action_count 94195455013925 step 0 next 2387
    2021-08-09 03:52:38.084934: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5d00 of size 256 by op assert_less_equal_20/Assert/AssertGuard/pivot_t/_574 action_count 94195455013926 step 0 next 2328
    2021-08-09 03:52:38.084956: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5e00 of size 256 by op assert_greater_equal_21/Assert/AssertGuard/pivot_f/_591 action_count 94195455013927 step 0 next 2270
    2021-08-09 03:52:38.084977: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd5f00 of size 256 by op assert_greater_equal_21/Assert/AssertGuard/pivot_t/_592 action_count 94195455013928 step 0 next 2252
    2021-08-09 03:52:38.085000: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6000 of size 256 by op gradient_tape/unet_depth3/encode2/batch_normalization_550/moments/scalar action_count 94195455013929 step 0 nex
    t 2392
    2021-08-09 03:52:38.085023: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6100 of size 256 by op assert_less_equal_21/Assert/AssertGuard/pivot_f/_601 action_count 94195455013930 step 0 next 2368
    2021-08-09 03:52:38.085044: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6200 of size 256 by op assert_less_equal_21/Assert/AssertGuard/pivot_t/_602 action_count 94195455013931 step 0 next 2385
    2021-08-09 03:52:38.085067: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6300 of size 256 by op assert_greater_equal_22/Assert/AssertGuard/pivot_f/_619 action_count 94195455013932 step 0 next 1296
    2021-08-09 03:52:38.085088: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6400 of size 256 by op assert_greater_equal_22/Assert/AssertGuard/pivot_t/_620 action_count 94195455013933 step 0 next 2303
    2021-08-09 03:52:38.085105: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6500 of size 256 by op assert_less_equal_22/Assert/AssertGuard/pivot_f/_629 action_count 94195455013934 step 0 next 2357
    2021-08-09 03:52:38.085124: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6600 of size 256 by op assert_less_equal_22/Assert/AssertGuard/pivot_t/_630 action_count 94195455013935 step 0 next 2273
    2021-08-09 03:52:38.085148: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6700 of size 256 by op assert_greater_equal_23/Assert/AssertGuard/pivot_f/_647 action_count 94195455013936 step 0 next 2365
    2021-08-09 03:52:38.085170: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6800 of size 256 by op assert_greater_equal_23/Assert/AssertGuard/pivot_t/_648 action_count 94195455013937 step 0 next 517
    2021-08-09 03:52:38.085191: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6900 of size 256 by op gradient_tape/unet_depth3/encode1/batch_normalization_549/moments/scalar action_count 94195455013938 step 0 nex
    t 2242
    2021-08-09 03:52:38.085214: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6a00 of size 256 by op assert_less_equal_23/Assert/AssertGuard/pivot_f/_657 action_count 94195455013939 step 0 next 1406
    2021-08-09 03:52:38.085234: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6b00 of size 256 by op assert_less_equal_23/Assert/AssertGuard/pivot_t/_658 action_count 94195455013940 step 0 next 1781
    2021-08-09 03:52:38.085257: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6c00 of size 256 by op assert_greater_equal_24/Assert/AssertGuard/pivot_f/_675 action_count 94195455013941 step 0 next 1698
    2021-08-09 03:52:38.085277: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6d00 of size 256 by op assert_greater_equal_24/Assert/AssertGuard/pivot_t/_676 action_count 94195455013942 step 0 next 2302
    2021-08-09 03:52:38.085296: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6e00 of size 256 by op assert_less_equal_24/Assert/AssertGuard/pivot_f/_685 action_count 94195455013943 step 0 next 468
    2021-08-09 03:52:38.085317: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd6f00 of size 256 by op assert_less_equal_24/Assert/AssertGuard/pivot_t/_686 action_count 94195455013944 step 0 next 2380
    2021-08-09 03:52:38.085338: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7000 of size 256 by op gradient_tape/unet_depth3/encode1/batch_normalization_548/moments/scalar action_count 94195455013945 step 0 nex
    t 2277
    2021-08-09 03:52:38.085357: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7100 of size 256 by op assert_greater_equal_25/Assert/AssertGuard/pivot_f/_703 action_count 94195455013946 step 0 next 1686
    2021-08-09 03:52:38.085379: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7200 of size 256 by op assert_greater_equal_25/Assert/AssertGuard/pivot_t/_704 action_count 94195455013947 step 0 next 2362
    2021-08-09 03:52:38.085397: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7300 of size 256 by op assert_less_equal_25/Assert/AssertGuard/pivot_f/_713 action_count 94195455013948 step 0 next 2245
    2021-08-09 03:52:38.085420: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7400 of size 256 by op assert_less_equal_25/Assert/AssertGuard/pivot_t/_714 action_count 94195455013949 step 0 next 2309
    2021-08-09 03:52:38.085444: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7500 of size 256 by op assert_greater_equal_26/Assert/AssertGuard/pivot_f/_731 action_count 94195455013950 step 0 next 2461
    2021-08-09 03:52:38.085465: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7600 of size 256 by op assert_greater_equal_26/Assert/AssertGuard/pivot_t/_732 action_count 94195455013951 step 0 next 2377
    2021-08-09 03:52:38.085486: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7700 of size 256 by op assert_less_equal_26/Assert/AssertGuard/pivot_f/_741 action_count 94195455013952 step 0 next 2298
    2021-08-09 03:52:38.085509: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7800 of size 256 by op assert_less_equal_26/Assert/AssertGuard/pivot_t/_742 action_count 94195455013953 step 0 next 2219
    2021-08-09 03:52:38.085535: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7900 of size 256 by op gradient_tape/unet_depth3/encode0/batch_normalization_547/moments/scalar action_count 94195455013954 step 0 nex
    t 1821
    2021-08-09 03:52:38.085555: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7a00 of size 256 by op assert_greater_equal_27/Assert/AssertGuard/pivot_f/_759 action_count 94195455013955 step 0 next 1829
    2021-08-09 03:52:38.085577: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7b00 of size 256 by op assert_greater_equal_27/Assert/AssertGuard/pivot_t/_760 action_count 94195455013956 step 0 next 2396
    2021-08-09 03:52:38.085598: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7c00 of size 256 by op assert_less_equal_27/Assert/AssertGuard/pivot_f/_769 action_count 94195455013957 step 0 next 2255
    2021-08-09 03:52:38.085620: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7d00 of size 256 by op assert_less_equal_27/Assert/AssertGuard/pivot_t/_770 action_count 94195455013958 step 0 next 573
    2021-08-09 03:52:38.085641: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7e00 of size 256 by op Sub action_count 94195065726203 step 0 next 63
    2021-08-09 03:52:38.085658: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd7f00 of size 256 by op Sub action_count 94195085385179 step 0 next 1215
    2021-08-09 03:52:38.085675: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd8000 of size 256 by op Sub action_count 94195085385180 step 0 next 631
    2021-08-09 03:52:38.085696: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd8100 of size 256 by op Sub action_count 94195094149558 step 0 next 1733
    2021-08-09 03:52:38.085716: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd8200 of size 256 by op Sub action_count 94195094149559 step 0 next 1648
    2021-08-09 03:52:38.085736: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd8300 of size 1792 by op Fill action_count 94195455013451 step 0 next 430
    2021-08-09 03:52:38.085753: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd8a00 of size 256 by op Sub action_count 94195065726216 step 0 next 425
    2021-08-09 03:52:38.085770: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd8b00 of size 512 by op Fill action_count 94195455013383 step 0 next 446
    2021-08-09 03:52:38.085790: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd8d00 of size 256 by op Sub action_count 94195065726217 step 0 next 1124
    2021-08-09 03:52:38.085810: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd8e00 of size 256 by op Sub action_count 94195085385138 step 0 next 1065
    2021-08-09 03:52:38.085830: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd8f00 of size 256 by op Sub action_count 94195065726230 step 0 next 921
    2021-08-09 03:52:38.085847: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9000 of size 256 by op Sub action_count 94195094149517 step 0 next 1149
    2021-08-09 03:52:38.085868: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9100 of size 256 by op assert_less_equal_17/Assert/AssertGuard/pivot_f/_489 action_count 94195467563408 step 0 next 2477
    2021-08-09 03:52:38.085889: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9200 of size 256 by op assert_less_equal_17/Assert/AssertGuard/pivot_t/_490 action_count 94195467563409 step 0 next 2456
    2021-08-09 03:52:38.085909: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9300 of size 256 by op gradient_tape/unet_depth3/two_conv_center/batch_normalization_569/moments/scalar action_count 94195467563410 st
    ep 0 next 1570
    2021-08-09 03:52:38.085930: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9400 of size 256 by op assert_greater_equal_18/Assert/AssertGuard/pivot_f/_507 action_count 94195467563411 step 0 next 2063
    2021-08-09 03:52:38.085950: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9500 of size 256 by op assert_greater_equal_18/Assert/AssertGuard/pivot_t/_508 action_count 94195467563412 step 0 next 2061
    2021-08-09 03:52:38.085968: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9600 of size 256 by op assert_less_equal_18/Assert/AssertGuard/pivot_f/_517 action_count 94195467563413 step 0 next 244
    2021-08-09 03:52:38.085989: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9700 of size 256 by op assert_less_equal_18/Assert/AssertGuard/pivot_t/_518 action_count 94195467563414 step 0 next 305
    2021-08-09 03:52:38.086010: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9800 of size 256 by op assert_greater_equal_19/Assert/AssertGuard/pivot_f/_535 action_count 94195467563415 step 0 next 317
    2021-08-09 03:52:38.086027: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9900 of size 256 by op assert_greater_equal_19/Assert/AssertGuard/pivot_t/_536 action_count 94195467563416 step 0 next 1228
    2021-08-09 03:52:38.086045: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9a00 of size 256 by op assert_less_equal_19/Assert/AssertGuard/pivot_f/_545 action_count 94195467563417 step 0 next 746
    2021-08-09 03:52:38.086063: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9b00 of size 256 by op assert_less_equal_19/Assert/AssertGuard/pivot_t/_546 action_count 94195467563418 step 0 next 979
    2021-08-09 03:52:38.086084: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9c00 of size 256 by op gradient_tape/unet_depth3/encode2/batch_normalization_568/moments/scalar action_count 94195467563419 step 0 nex
    t 1375
    2021-08-09 03:52:38.086104: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9d00 of size 256 by op assert_greater_equal_20/Assert/AssertGuard/pivot_f/_563 action_count 94195467563420 step 0 next 1207
    2021-08-09 03:52:38.086125: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9e00 of size 256 by op assert_greater_equal_20/Assert/AssertGuard/pivot_t/_564 action_count 94195467563421 step 0 next 1117
    2021-08-09 03:52:38.086148: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfd9f00 of size 256 by op assert_less_equal_20/Assert/AssertGuard/pivot_f/_573 action_count 94195467563422 step 0 next 959
    2021-08-09 03:52:38.086170: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfda000 of size 256 by op assert_less_equal_20/Assert/AssertGuard/pivot_t/_574 action_count 94195467563423 step 0 next 1059
    2021-08-09 03:52:38.086191: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfda100 of size 256 by op assert_greater_equal_21/Assert/AssertGuard/pivot_f/_591 action_count 94195467563424 step 0 next 246
    2021-08-09 03:52:38.086213: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfda200 of size 256 by op assert_greater_equal_21/Assert/AssertGuard/pivot_t/_592 action_count 94195467563425 step 0 next 2589
    2021-08-09 03:52:38.086235: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfda300 of size 256 by op gradient_tape/unet_depth3/encode2/batch_normalization_567/moments/scalar action_count 94195467563426 step 0 nex
    t 2465
    2021-08-09 03:52:38.086257: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfda400 of size 256 by op assert_less_equal_21/Assert/AssertGuard/pivot_f/_601 action_count 94195467563427 step 0 next 175
    2021-08-09 03:52:38.086274: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfda500 of size 256 by op assert_less_equal_21/Assert/AssertGuard/pivot_t/_602 action_count 94195467563428 step 0 next 1502
    2021-08-09 03:52:38.086295: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfda600 of size 256 by op assert_greater_equal_22/Assert/AssertGuard/pivot_f/_619 action_count 94195467563429 step 0 next 1057
    2021-08-09 03:52:38.086314: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfda700 of size 256 by op assert_greater_equal_22/Assert/AssertGuard/pivot_t/_620 action_count 94195467563430 step 0 next 173
    2021-08-09 03:52:38.086338: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfda800 of size 256 by op assert_less_equal_22/Assert/AssertGuard/pivot_f/_629 action_count 94195467563431 step 0 next 148
    2021-08-09 03:52:38.086360: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfda900 of size 256 by op assert_less_equal_22/Assert/AssertGuard/pivot_t/_630 action_count 94195467563432 step 0 next 532
    2021-08-09 03:52:38.086380: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdaa00 of size 256 by op assert_greater_equal_23/Assert/AssertGuard/pivot_f/_647 action_count 94195467563433 step 0 next 2431
    2021-08-09 03:52:38.086401: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdab00 of size 256 by op assert_greater_equal_23/Assert/AssertGuard/pivot_t/_648 action_count 94195467563434 step 0 next 288
    2021-08-09 03:52:38.086424: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdac00 of size 256 by op gradient_tape/unet_depth3/encode1/batch_normalization_566/moments/scalar action_count 94195467563435 step 0 nex
    t 1144
    2021-08-09 03:52:38.086444: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdad00 of size 256 by op assert_less_equal_23/Assert/AssertGuard/pivot_f/_657 action_count 94195467563436 step 0 next 1242
    2021-08-09 03:52:38.086467: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdae00 of size 256 by op assert_less_equal_23/Assert/AssertGuard/pivot_t/_658 action_count 94195467563437 step 0 next 774
    2021-08-09 03:52:38.086489: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdaf00 of size 256 by op assert_greater_equal_24/Assert/AssertGuard/pivot_f/_675 action_count 94195467563438 step 0 next 593
    2021-08-09 03:52:38.086507: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdb000 of size 256 by op assert_greater_equal_24/Assert/AssertGuard/pivot_t/_676 action_count 94195467563439 step 0 next 1856
    2021-08-09 03:52:38.086533: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdb100 of size 256 by op assert_less_equal_24/Assert/AssertGuard/pivot_f/_685 action_count 94195467563440 step 0 next 1759
    2021-08-09 03:52:38.086551: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdb200 of size 256 by op assert_less_equal_24/Assert/AssertGuard/pivot_t/_686 action_count 94195467563441 step 0 next 86
    2021-08-09 03:52:38.086574: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdb300 of size 256 by op gradient_tape/unet_depth3/encode1/batch_normalization_565/moments/scalar action_count 94195467563442 step 0 nex
    t 295
    2021-08-09 03:52:38.086595: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdb400 of size 256 by op assert_greater_equal_25/Assert/AssertGuard/pivot_f/_703 action_count 94195467563443 step 0 next 2052
    2021-08-09 03:52:38.086615: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdb500 of size 256 by op assert_greater_equal_25/Assert/AssertGuard/pivot_t/_704 action_count 94195467563444 step 0 next 1063
    2021-08-09 03:52:38.086633: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdb600 of size 256 by op assert_less_equal_25/Assert/AssertGuard/pivot_f/_713 action_count 94195467563445 step 0 next 2449
    2021-08-09 03:52:38.086653: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdb700 of size 256 by op assert_less_equal_25/Assert/AssertGuard/pivot_t/_714 action_count 94195467563446 step 0 next 2414
    2021-08-09 03:52:38.086671: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdb800 of size 256 by op assert_greater_equal_26/Assert/AssertGuard/pivot_f/_731 action_count 94195467563447 step 0 next 1213
    2021-08-09 03:52:38.086691: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdb900 of size 256 by op assert_greater_equal_26/Assert/AssertGuard/pivot_t/_732 action_count 94195467563448 step 0 next 2266
    2021-08-09 03:52:38.086712: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdba00 of size 256 by op assert_less_equal_26/Assert/AssertGuard/pivot_f/_741 action_count 94195467563449 step 0 next 866
    2021-08-09 03:52:38.086732: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdbb00 of size 256 by op assert_less_equal_26/Assert/AssertGuard/pivot_t/_742 action_count 94195467563450 step 0 next 42
    2021-08-09 03:52:38.086753: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdbc00 of size 256 by op gradient_tape/unet_depth3/encode0/batch_normalization_564/moments/scalar action_count 94195467563451 step 0 nex
    t 935
    2021-08-09 03:52:38.086775: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdbd00 of size 256 by op assert_greater_equal_27/Assert/AssertGuard/pivot_f/_759 action_count 94195467563452 step 0 next 929
    2021-08-09 03:52:38.086792: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdbe00 of size 256 by op assert_greater_equal_27/Assert/AssertGuard/pivot_t/_760 action_count 94195467563453 step 0 next 533
    2021-08-09 03:52:38.086814: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdbf00 of size 256 by op assert_less_equal_27/Assert/AssertGuard/pivot_f/_769 action_count 94195467563454 step 0 next 763
    2021-08-09 03:52:38.086838: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdc000 of size 256 by op assert_less_equal_27/Assert/AssertGuard/pivot_t/_770 action_count 94195467563455 step 0 next 1804
    2021-08-09 03:52:38.086859: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdc100 of size 256 by op assert_greater_equal_28/Assert/AssertGuard/pivot_f/_787 action_count 94195467563456 step 0 next 1857
    2021-08-09 03:52:38.086880: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdc200 of size 256 by op assert_greater_equal_28/Assert/AssertGuard/pivot_t/_788 action_count 94195467563457 step 0 next 1084
    2021-08-09 03:52:38.086902: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdc300 of size 256 by op assert_less_equal_28/Assert/AssertGuard/pivot_f/_797 action_count 94195467563458 step 0 next 1122
    2021-08-09 03:52:38.086920: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdc400 of size 256 by op assert_less_equal_28/Assert/AssertGuard/pivot_t/_798 action_count 94195467563459 step 0 next 2180
    2021-08-09 03:52:38.086941: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdc500 of size 256 by op gradient_tape/unet_depth3/encode0/batch_normalization_563/moments/scalar action_count 94195467563460 step 0 nex
    t 1080
    2021-08-09 03:52:38.086962: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdc600 of size 256 by op assert_greater_equal_29/Assert/AssertGuard/pivot_f/_815 action_count 94195467563461 step 0 next 89
    2021-08-09 03:52:38.086979: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdc700 of size 256 by op assert_greater_equal_29/Assert/AssertGuard/pivot_t/_816 action_count 94195467563462 step 0 next 2442
    2021-08-09 03:52:38.087001: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdc800 of size 256 by op Adam/Adam/Const action_count 94195467563463 step 0 next 1158
    2021-08-09 03:52:38.087023: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdc900 of size 256 by op assert_less_equal_29/Assert/AssertGuard/pivot_f/_825 action_count 94195467563464 step 0 next 580
    2021-08-09 03:52:38.087041: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdca00 of size 256 by op assert_less_equal_29/Assert/AssertGuard/pivot_t/_826 action_count 94195467563465 step 0 next 71
    2021-08-09 03:52:38.087058: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdcb00 of size 256 by op assert_greater_equal_30/Assert/AssertGuard/pivot_f/_843 action_count 94195467563466 step 0 next 993
    2021-08-09 03:52:38.087076: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdcc00 of size 256 by op assert_greater_equal_30/Assert/AssertGuard/pivot_t/_844 action_count 94195467563467 step 0 next 1831
    2021-08-09 03:52:38.087098: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdcd00 of size 256 by op assert_less_equal_30/Assert/AssertGuard/pivot_f/_853 action_count 94195467563468 step 0 next 1884
    2021-08-09 03:52:38.087118: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdce00 of size 256 by op assert_less_equal_30/Assert/AssertGuard/pivot_t/_854 action_count 94195467563469 step 0 next 1294
    2021-08-09 03:52:38.087139: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdcf00 of size 256 by op assert_greater_equal_31/Assert/AssertGuard/pivot_f/_871 action_count 94195467563470 step 0 next 1901
    2021-08-09 03:52:38.087159: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdd000 of size 256 by op assert_greater_equal_31/Assert/AssertGuard/pivot_t/_872 action_count 94195467563471 step 0 next 331
    2021-08-09 03:52:38.087178: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdd100 of size 256 by op assert_less_equal_31/Assert/AssertGuard/pivot_f/_881 action_count 94195467563472 step 0 next 1340
    2021-08-09 03:52:38.087199: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdd200 of size 256 by op assert_less_equal_31/Assert/AssertGuard/pivot_t/_882 action_count 94195467563473 step 0 next 208
    2021-08-09 03:52:38.087217: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdd300 of size 256 by op assert_greater_equal_32/Assert/AssertGuard/pivot_f/_899 action_count 94195467563474 step 0 next 2434
    2021-08-09 03:52:38.087234: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdd400 of size 256 by op assert_greater_equal_32/Assert/AssertGuard/pivot_t/_900 action_count 94195467563475 step 0 next 287
    2021-08-09 03:52:38.087251: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdd500 of size 256 by op assert_less_equal_32/Assert/AssertGuard/pivot_f/_909 action_count 94195467563476 step 0 next 745
    2021-08-09 03:52:38.087273: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdd600 of size 256 by op assert_less_equal_32/Assert/AssertGuard/pivot_t/_910 action_count 94195467563477 step 0 next 206
    2021-08-09 03:52:38.087297: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdd700 of size 256 by op assert_greater_equal_5/Assert/AssertGuard/pivot_f/_143 action_count 94195468133277 step 0 next 455
    2021-08-09 03:52:38.087314: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdd800 of size 256 by op assert_greater_equal_5/Assert/AssertGuard/pivot_t/_144 action_count 94195468133278 step 0 next 400
    2021-08-09 03:52:38.087336: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdd900 of size 256 by op assert_less_equal_5/Assert/AssertGuard/pivot_f/_153 action_count 94195468133279 step 0 next 1081
    2021-08-09 03:52:38.087357: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdda00 of size 256 by op assert_less_equal_5/Assert/AssertGuard/pivot_t/_154 action_count 94195468133280 step 0 next 2184
    2021-08-09 03:52:38.087380: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfddb00 of size 256 by op assert_greater_equal_6/Assert/AssertGuard/pivot_f/_171 action_count 94195468133281 step 0 next 2247
    2021-08-09 03:52:38.087401: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfddc00 of size 256 by op assert_greater_equal_6/Assert/AssertGuard/pivot_t/_172 action_count 94195468133282 step 0 next 1551
    2021-08-09 03:52:38.087422: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfddd00 of size 256 by op assert_less_equal_6/Assert/AssertGuard/pivot_f/_181 action_count 94195468133283 step 0 next 2030
    2021-08-09 03:52:38.087443: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdde00 of size 256 by op assert_less_equal_6/Assert/AssertGuard/pivot_t/_182 action_count 94195468133284 step 0 next 1665
    2021-08-09 03:52:38.087460: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfddf00 of size 256 by op assert_greater_equal_7/Assert/AssertGuard/pivot_f/_199 action_count 94195468133285 step 0 next 1352
    2021-08-09 03:52:38.087483: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfde000 of size 256 by op assert_greater_equal_7/Assert/AssertGuard/pivot_t/_200 action_count 94195468133286 step 0 next 714
    2021-08-09 03:52:38.087506: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfde100 of size 256 by op assert_less_equal_7/Assert/AssertGuard/pivot_f/_209 action_count 94195468133287 step 0 next 149
    2021-08-09 03:52:38.087528: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfde200 of size 256 by op assert_less_equal_7/Assert/AssertGuard/pivot_t/_210 action_count 94195468133288 step 0 next 2132
    2021-08-09 03:52:38.087546: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfde300 of size 256 by op assert_greater_equal_8/Assert/AssertGuard/pivot_f/_227 action_count 94195468133289 step 0 next 374
    2021-08-09 03:52:38.087564: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfde400 of size 256 by op assert_greater_equal_8/Assert/AssertGuard/pivot_t/_228 action_count 94195468133290 step 0 next 1025
    2021-08-09 03:52:38.087588: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfde500 of size 256 by op assert_less_equal_8/Assert/AssertGuard/pivot_f/_237 action_count 94195468133291 step 0 next 2645
    2021-08-09 03:52:38.087609: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfde600 of size 256 by op assert_less_equal_8/Assert/AssertGuard/pivot_t/_238 action_count 94195468133292 step 0 next 2179
    2021-08-09 03:52:38.087630: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfde700 of size 256 by op assert_greater_equal_9/Assert/AssertGuard/pivot_f/_255 action_count 94195468133293 step 0 next 800
    2021-08-09 03:52:38.087649: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfde800 of size 256 by op assert_greater_equal_9/Assert/AssertGuard/pivot_t/_256 action_count 94195468133294 step 0 next 2136
    2021-08-09 03:52:38.087672: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfde900 of size 256 by op assert_less_equal_9/Assert/AssertGuard/pivot_f/_265 action_count 94195468133295 step 0 next 1320
    2021-08-09 03:52:38.087695: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdea00 of size 256 by op assert_less_equal_9/Assert/AssertGuard/pivot_t/_266 action_count 94195468133296 step 0 next 1892
    2021-08-09 03:52:38.087717: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdeb00 of size 256 by op assert_greater_equal_10/Assert/AssertGuard/pivot_f/_283 action_count 94195468133297 step 0 next 1601
    2021-08-09 03:52:38.087736: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdec00 of size 256 by op assert_greater_equal_10/Assert/AssertGuard/pivot_t/_284 action_count 94195468133298 step 0 next 196
    2021-08-09 03:52:38.087754: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfded00 of size 256 by op assert_less_equal_10/Assert/AssertGuard/pivot_f/_293 action_count 94195468133299 step 0 next 591
    2021-08-09 03:52:38.087776: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdee00 of size 256 by op assert_less_equal_10/Assert/AssertGuard/pivot_t/_294 action_count 94195468133300 step 0 next 396
    2021-08-09 03:52:38.087799: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdef00 of size 256 by op assert_greater_equal_11/Assert/AssertGuard/pivot_f/_311 action_count 94195468133301 step 0 next 1814
    2021-08-09 03:52:38.087820: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdf000 of size 256 by op assert_greater_equal_11/Assert/AssertGuard/pivot_t/_312 action_count 94195468133302 step 0 next 1236
    2021-08-09 03:52:38.087843: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdf100 of size 256 by op assert_less_equal_11/Assert/AssertGuard/pivot_f/_321 action_count 94195468133303 step 0 next 2579
    2021-08-09 03:52:38.087863: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdf200 of size 256 by op assert_less_equal_11/Assert/AssertGuard/pivot_t/_322 action_count 94195468133304 step 0 next 2426
    2021-08-09 03:52:38.087884: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdf300 of size 256 by op assert_greater_equal_12/Assert/AssertGuard/pivot_f/_339 action_count 94195468133305 step 0 next 1668
    2021-08-09 03:52:38.087904: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdf400 of size 256 by op assert_greater_equal_12/Assert/AssertGuard/pivot_t/_340 action_count 94195468133306 step 0 next 1310
    2021-08-09 03:52:38.087928: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdf500 of size 256 by op assert_less_equal_12/Assert/AssertGuard/pivot_f/_349 action_count 94195468133307 step 0 next 479
    2021-08-09 03:52:38.087945: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdf600 of size 256 by op assert_less_equal_12/Assert/AssertGuard/pivot_t/_350 action_count 94195468133308 step 0 next 19
    2021-08-09 03:52:38.087968: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdf700 of size 256 by op assert_greater_equal_13/Assert/AssertGuard/pivot_f/_367 action_count 94195468133309 step 0 next 2079
    2021-08-09 03:52:38.087988: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdf800 of size 256 by op assert_greater_equal_13/Assert/AssertGuard/pivot_t/_368 action_count 94195468133310 step 0 next 585
    2021-08-09 03:52:38.088011: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdf900 of size 256 by op assert_less_equal_13/Assert/AssertGuard/pivot_f/_377 action_count 94195468133311 step 0 next 1988
    2021-08-09 03:52:38.088032: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdfa00 of size 256 by op assert_less_equal_13/Assert/AssertGuard/pivot_t/_378 action_count 94195468133312 step 0 next 1580
    2021-08-09 03:52:38.088054: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdfb00 of size 256 by op assert_greater_equal_14/Assert/AssertGuard/pivot_f/_395 action_count 94195468133313 step 0 next 277
    2021-08-09 03:52:38.088072: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdfc00 of size 256 by op assert_greater_equal_14/Assert/AssertGuard/pivot_t/_396 action_count 94195468133314 step 0 next 2467
    2021-08-09 03:52:38.088089: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdfd00 of size 256 by op assert_less_equal_14/Assert/AssertGuard/pivot_f/_405 action_count 94195468133315 step 0 next 870
    2021-08-09 03:52:38.088109: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdfe00 of size 256 by op assert_less_equal_14/Assert/AssertGuard/pivot_t/_406 action_count 94195468133316 step 0 next 702
    2021-08-09 03:52:38.088130: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfdff00 of size 256 by op assert_greater_equal_15/Assert/AssertGuard/pivot_f/_423 action_count 94195468133317 step 0 next 2279
    2021-08-09 03:52:38.088152: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0000 of size 256 by op assert_greater_equal_15/Assert/AssertGuard/pivot_t/_424 action_count 94195468133318 step 0 next 452
    2021-08-09 03:52:38.088173: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0100 of size 256 by op assert_less_equal_15/Assert/AssertGuard/pivot_f/_433 action_count 94195468133319 step 0 next 1777
    2021-08-09 03:52:38.088194: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0200 of size 256 by op assert_less_equal_15/Assert/AssertGuard/pivot_t/_434 action_count 94195468133320 step 0 next 974
    2021-08-09 03:52:38.088213: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0300 of size 256 by op assert_greater_equal_16/Assert/AssertGuard/pivot_f/_451 action_count 94195468133321 step 0 next 831
    2021-08-09 03:52:38.088235: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0400 of size 256 by op assert_greater_equal_16/Assert/AssertGuard/pivot_t/_452 action_count 94195468133322 step 0 next 302
    2021-08-09 03:52:38.088252: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0500 of size 256 by op assert_less_equal_16/Assert/AssertGuard/pivot_f/_461 action_count 94195468133323 step 0 next 1034
    2021-08-09 03:52:38.088270: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0600 of size 256 by op assert_less_equal_16/Assert/AssertGuard/pivot_t/_462 action_count 94195468133324 step 0 next 1976
    2021-08-09 03:52:38.088293: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0700 of size 256 by op assert_greater_equal_17/Assert/AssertGuard/pivot_f/_479 action_count 94195468133325 step 0 next 660
    2021-08-09 03:52:38.088314: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0800 of size 256 by op assert_greater_equal_17/Assert/AssertGuard/pivot_t/_480 action_count 94195468133326 step 0 next 618
    2021-08-09 03:52:38.088337: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0900 of size 256 by op assert_less_equal_17/Assert/AssertGuard/pivot_f/_489 action_count 94195468133327 step 0 next 1226
    2021-08-09 03:52:38.088360: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0a00 of size 256 by op assert_less_equal_17/Assert/AssertGuard/pivot_t/_490 action_count 94195468133328 step 0 next 688
    2021-08-09 03:52:38.088383: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0b00 of size 256 by op assert_greater_equal_18/Assert/AssertGuard/pivot_f/_507 action_count 94195468133329 step 0 next 1711
    2021-08-09 03:52:38.088405: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0c00 of size 256 by op assert_greater_equal_18/Assert/AssertGuard/pivot_t/_508 action_count 94195468133330 step 0 next 1005
    2021-08-09 03:52:38.088425: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0d00 of size 256 by op assert_less_equal_18/Assert/AssertGuard/pivot_f/_517 action_count 94195468133331 step 0 next 77
    2021-08-09 03:52:38.088445: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0e00 of size 256 by op assert_less_equal_18/Assert/AssertGuard/pivot_t/_518 action_count 94195468133332 step 0 next 639
    2021-08-09 03:52:38.088466: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe0f00 of size 256 by op assert_greater_equal_19/Assert/AssertGuard/pivot_f/_535 action_count 94195468133333 step 0 next 1977
    2021-08-09 03:52:38.088484: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1000 of size 256 by op assert_greater_equal_19/Assert/AssertGuard/pivot_t/_536 action_count 94195468133334 step 0 next 190
    2021-08-09 03:52:38.088505: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1100 of size 256 by op assert_less_equal_19/Assert/AssertGuard/pivot_f/_545 action_count 94195468133335 step 0 next 476
    2021-08-09 03:52:38.088539: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1200 of size 256 by op assert_less_equal_19/Assert/AssertGuard/pivot_t/_546 action_count 94195468133336 step 0 next 923
    2021-08-09 03:52:38.088580: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1300 of size 256 by op assert_greater_equal_20/Assert/AssertGuard/pivot_f/_563 action_count 94195468133337 step 0 next 624
    2021-08-09 03:52:38.088613: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1400 of size 256 by op assert_greater_equal_20/Assert/AssertGuard/pivot_t/_564 action_count 94195468133338 step 0 next 1916
    2021-08-09 03:52:38.088638: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1500 of size 256 by op assert_less_equal_20/Assert/AssertGuard/pivot_f/_573 action_count 94195468133339 step 0 next 1308
    2021-08-09 03:52:38.088661: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1600 of size 256 by op assert_less_equal_20/Assert/AssertGuard/pivot_t/_574 action_count 94195468133340 step 0 next 251
    2021-08-09 03:52:38.088682: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1700 of size 256 by op assert_greater_equal_21/Assert/AssertGuard/pivot_f/_591 action_count 94195468133341 step 0 next 2471
    2021-08-09 03:52:38.088703: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1800 of size 256 by op assert_greater_equal_21/Assert/AssertGuard/pivot_t/_592 action_count 94195468133342 step 0 next 35
    2021-08-09 03:52:38.088725: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1900 of size 256 by op assert_less_equal_21/Assert/AssertGuard/pivot_f/_601 action_count 94195468133343 step 0 next 2394
    2021-08-09 03:52:38.088743: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1a00 of size 256 by op assert_less_equal_21/Assert/AssertGuard/pivot_t/_602 action_count 94195468133344 step 0 next 880
    2021-08-09 03:52:38.088765: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1b00 of size 256 by op assert_greater_equal_22/Assert/AssertGuard/pivot_f/_619 action_count 94195468133345 step 0 next 977
    2021-08-09 03:52:38.088787: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1c00 of size 256 by op assert_greater_equal_22/Assert/AssertGuard/pivot_t/_620 action_count 94195468133346 step 0 next 682
    2021-08-09 03:52:38.088809: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1d00 of size 256 by op assert_less_equal_22/Assert/AssertGuard/pivot_f/_629 action_count 94195468133347 step 0 next 1531
    2021-08-09 03:52:38.088827: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1e00 of size 256 by op assert_less_equal_22/Assert/AssertGuard/pivot_t/_630 action_count 94195468133348 step 0 next 904
    2021-08-09 03:52:38.088850: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe1f00 of size 256 by op assert_greater_equal_23/Assert/AssertGuard/pivot_f/_647 action_count 94195468133349 step 0 next 1524
    2021-08-09 03:52:38.088872: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2000 of size 256 by op assert_greater_equal_23/Assert/AssertGuard/pivot_t/_648 action_count 94195468133350 step 0 next 1713
    2021-08-09 03:52:38.088893: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2100 of size 256 by op assert_less_equal_23/Assert/AssertGuard/pivot_f/_657 action_count 94195468133351 step 0 next 829
    2021-08-09 03:52:38.088913: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2200 of size 256 by op assert_less_equal_23/Assert/AssertGuard/pivot_t/_658 action_count 94195468133352 step 0 next 1533
    2021-08-09 03:52:38.088932: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2300 of size 256 by op assert_greater_equal_24/Assert/AssertGuard/pivot_f/_675 action_count 94195468133353 step 0 next 1279
    2021-08-09 03:52:38.088954: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2400 of size 256 by op assert_greater_equal_24/Assert/AssertGuard/pivot_t/_676 action_count 94195468133354 step 0 next 1639
    2021-08-09 03:52:38.088977: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2500 of size 256 by op assert_less_equal_24/Assert/AssertGuard/pivot_f/_685 action_count 94195468133355 step 0 next 179
    2021-08-09 03:52:38.088998: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2600 of size 256 by op assert_less_equal_24/Assert/AssertGuard/pivot_t/_686 action_count 94195468133356 step 0 next 1984
    2021-08-09 03:52:38.089019: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2700 of size 256 by op assert_greater_equal_25/Assert/AssertGuard/pivot_f/_703 action_count 94195468133357 step 0 next 1918
    2021-08-09 03:52:38.089040: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2800 of size 256 by op assert_greater_equal_25/Assert/AssertGuard/pivot_t/_704 action_count 94195468133358 step 0 next 329
    2021-08-09 03:52:38.089059: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2900 of size 256 by op assert_less_equal_25/Assert/AssertGuard/pivot_f/_713 action_count 94195468133359 step 0 next 2060
    2021-08-09 03:52:38.089083: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2a00 of size 256 by op assert_less_equal_25/Assert/AssertGuard/pivot_t/_714 action_count 94195468133360 step 0 next 1820
    2021-08-09 03:52:38.089100: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2b00 of size 256 by op assert_greater_equal_26/Assert/AssertGuard/pivot_f/_731 action_count 94195468133361 step 0 next 1874
    2021-08-09 03:52:38.089122: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2c00 of size 256 by op assert_greater_equal_26/Assert/AssertGuard/pivot_t/_732 action_count 94195468133362 step 0 next 1408
    2021-08-09 03:52:38.089154: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2d00 of size 256 by op assert_less_equal_26/Assert/AssertGuard/pivot_f/_741 action_count 94195468133363 step 0 next 584
    2021-08-09 03:52:38.089170: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2e00 of size 256 by op assert_less_equal_26/Assert/AssertGuard/pivot_t/_742 action_count 94195468133364 step 0 next 1372
    2021-08-09 03:52:38.089192: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe2f00 of size 256 by op assert_greater_equal_27/Assert/AssertGuard/pivot_f/_759 action_count 94195468133365 step 0 next 43
    2021-08-09 03:52:38.089213: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3000 of size 256 by op assert_greater_equal_27/Assert/AssertGuard/pivot_t/_760 action_count 94195468133366 step 0 next 1041
    2021-08-09 03:52:38.089233: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3100 of size 256 by op assert_less_equal_27/Assert/AssertGuard/pivot_f/_769 action_count 94195468133367 step 0 next 1261
    2021-08-09 03:52:38.089254: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3200 of size 256 by op assert_less_equal_27/Assert/AssertGuard/pivot_t/_770 action_count 94195468133368 step 0 next 1939
    2021-08-09 03:52:38.089275: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3300 of size 256 by op assert_greater_equal_28/Assert/AssertGuard/pivot_f/_787 action_count 94195468133369 step 0 next 576
    2021-08-09 03:52:38.089295: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3400 of size 256 by op assert_greater_equal_28/Assert/AssertGuard/pivot_t/_788 action_count 94195468133370 step 0 next 1836
    2021-08-09 03:52:38.089315: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3500 of size 256 by op assert_less_equal_28/Assert/AssertGuard/pivot_f/_797 action_count 94195468133371 step 0 next 1749
    2021-08-09 03:52:38.089335: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3600 of size 256 by op assert_less_equal_28/Assert/AssertGuard/pivot_t/_798 action_count 94195468133372 step 0 next 36
    2021-08-09 03:52:38.089356: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3700 of size 256 by op assert_greater_equal_29/Assert/AssertGuard/pivot_f/_815 action_count 94195468133373 step 0 next 1938
    2021-08-09 03:52:38.089377: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3800 of size 256 by op assert_greater_equal_29/Assert/AssertGuard/pivot_t/_816 action_count 94195468133374 step 0 next 2130
    2021-08-09 03:52:38.089396: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3900 of size 256 by op assert_less_equal_29/Assert/AssertGuard/pivot_f/_825 action_count 94195468133375 step 0 next 2473
    2021-08-09 03:52:38.089416: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3a00 of size 256 by op assert_less_equal_29/Assert/AssertGuard/pivot_t/_826 action_count 94195468133376 step 0 next 348
    2021-08-09 03:52:38.089436: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3b00 of size 256 by op assert_greater_equal_30/Assert/AssertGuard/pivot_f/_843 action_count 94195468133377 step 0 next 1911
    2021-08-09 03:52:38.089455: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3c00 of size 256 by op assert_greater_equal_30/Assert/AssertGuard/pivot_t/_844 action_count 94195468133378 step 0 next 555
    2021-08-09 03:52:38.089476: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3d00 of size 256 by op assert_less_equal_30/Assert/AssertGuard/pivot_f/_853 action_count 94195468133379 step 0 next 2120
    2021-08-09 03:52:38.089497: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3e00 of size 256 by op assert_less_equal_30/Assert/AssertGuard/pivot_t/_854 action_count 94195468133380 step 0 next 543
    2021-08-09 03:52:38.089523: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe3f00 of size 256 by op assert_greater_equal_31/Assert/AssertGuard/pivot_f/_871 action_count 94195468133381 step 0 next 2594
    2021-08-09 03:52:38.089546: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4000 of size 256 by op assert_greater_equal_31/Assert/AssertGuard/pivot_t/_872 action_count 94195468133382 step 0 next 625
    2021-08-09 03:52:38.089565: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4100 of size 256 by op assert_less_equal_31/Assert/AssertGuard/pivot_f/_881 action_count 94195468133383 step 0 next 2083
    2021-08-09 03:52:38.089585: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4200 of size 256 by op assert_less_equal_31/Assert/AssertGuard/pivot_t/_882 action_count 94195468133384 step 0 next 239
    2021-08-09 03:52:38.089604: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4300 of size 256 by op assert_greater_equal_32/Assert/AssertGuard/pivot_f/_899 action_count 94195468133385 step 0 next 319
    2021-08-09 03:52:38.089624: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4400 of size 256 by op assert_greater_equal_32/Assert/AssertGuard/pivot_t/_900 action_count 94195468133386 step 0 next 327
    2021-08-09 03:52:38.089646: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4500 of size 256 by op assert_less_equal_32/Assert/AssertGuard/pivot_f/_909 action_count 94195468133387 step 0 next 93
    2021-08-09 03:52:38.089666: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4600 of size 256 by op assert_less_equal_32/Assert/AssertGuard/pivot_t/_910 action_count 94195468133388 step 0 next 1947
    2021-08-09 03:52:38.089685: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4700 of size 256 by op Sub action_count 94195480110842 step 0 next 540
    2021-08-09 03:52:38.089704: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4800 of size 256 by op AssignVariableOp action_count 94195480053210 step 0 next 316
    2021-08-09 03:52:38.089724: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4900 of size 256 by op AssignVariableOp action_count 94195480053212 step 0 next 2347
    2021-08-09 03:52:38.089743: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4a00 of size 512 by op Fill action_count 94195480110824 step 0 next 2410
    2021-08-09 03:52:38.089762: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4c00 of size 256 by op AssignVariableOp action_count 94195480053246 step 0 next 1466
    2021-08-09 03:52:38.089782: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4d00 of size 256 by op AssignVariableOp action_count 94195480053250 step 0 next 2149
    2021-08-09 03:52:38.089802: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4e00 of size 256 by op AssignVariableOp action_count 94195480053252 step 0 next 356
    2021-08-09 03:52:38.089821: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe4f00 of size 256 by op AssignVariableOp action_count 94195480053254 step 0 next 2000
    2021-08-09 03:52:38.089840: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe5000 of size 256 by op AssignVariableOp action_count 94195480053256 step 0 next 2212
    2021-08-09 03:52:38.089859: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe5100 of size 512 by op AssignVariableOp action_count 94195480053258 step 0 next 872
    2021-08-09 03:52:38.089879: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe5300 of size 512 by op AssignVariableOp action_count 94195480053262 step 0 next 23
    2021-08-09 03:52:38.089899: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe5500 of size 256 by op AssignVariableOp action_count 94195480053266 step 0 next 518
    2021-08-09 03:52:38.089918: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe5600 of size 256 by op AssignVariableOp action_count 94195480053268 step 0 next 1096
    2021-08-09 03:52:38.089937: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe5700 of size 256 by op AssignVariableOp action_count 94195480053270 step 0 next 2493
    2021-08-09 03:52:38.089956: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe5800 of size 256 by op AssignVariableOp action_count 94195480053272 step 0 next 2173
    2021-08-09 03:52:38.089975: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe5900 of size 512 by op AssignVariableOp action_count 94195480053260 step 0 next 1908
    2021-08-09 03:52:38.089995: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe5b00 of size 512 by op AssignVariableOp action_count 94195480053264 step 0 next 1606
    2021-08-09 03:52:38.090013: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe5d00 of size 512 by op Fill action_count 94195480110825 step 0 next 1046
    2021-08-09 03:52:38.090031: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe5f00 of size 256 by op AssignAddVariableOp_3 action_count 94195480109333 step 17368542405085625385 next 811
    2021-08-09 03:52:38.090050: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe6000 of size 256 by op Sub action_count 94195480110843 step 0 next 854
    2021-08-09 03:52:38.090070: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe6100 of size 256 by op AssignAddVariableOp_13 action_count 94195480109431 step 17368542405085625385 next 2268
    2021-08-09 03:52:38.090091: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe6200 of size 256 by op AssignAddVariableOp_19 action_count 94195480109375 step 17368542405085625385 next 1085
    2021-08-09 03:52:38.090112: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe6300 of size 256 by op AssignAddVariableOp_21 action_count 94195480109435 step 17368542405085625385 next 1013
    2021-08-09 03:52:38.090131: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe6400 of size 256 by op Sub action_count 94195094149710 step 0 next 1794
    2021-08-09 03:52:38.090150: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe6500 of size 256 by op Sub action_count 94195094149711 step 0 next 1075
    2021-08-09 03:52:38.090169: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe6600 of size 1792 by op Fill action_count 94195455013684 step 0 next 1463
    2021-08-09 03:52:38.090188: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe6d00 of size 1792 by op Fill action_count 94195455013686 step 0 next 401
    2021-08-09 03:52:38.090207: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe7400 of size 1792 by op Fill action_count 94195455013687 step 0 next 1373
    2021-08-09 03:52:38.090226: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe7b00 of size 2304 by op Fill action_count 94195455013688 step 0 next 1595
    2021-08-09 03:52:38.090244: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe8400 of size 256 by op Sub action_count 94195094149724 step 0 next 1589
    2021-08-09 03:52:38.090263: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe8500 of size 256 by op Sub action_count 94195094149725 step 0 next 1735
    2021-08-09 03:52:38.090282: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe8600 of size 2048 by op Fill action_count 94195467563205 step 0 next 1792
    2021-08-09 03:52:38.090302: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe8e00 of size 1792 by op Fill action_count 94195467563207 step 0 next 528
    2021-08-09 03:52:38.090323: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe9500 of size 1792 by op Fill action_count 94195467563208 step 0 next 711
    2021-08-09 03:52:38.090341: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfe9c00 of size 1792 by op Fill action_count 94195467563209 step 0 next 437
    2021-08-09 03:52:38.090362: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfea300 of size 1792 by op Fill action_count 94195467563211 step 0 next 2070
    2021-08-09 03:52:38.090381: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfeaa00 of size 1792 by op Fill action_count 94195467563212 step 0 next 2169
    2021-08-09 03:52:38.090400: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfeb100 of size 1792 by op Fill action_count 94195467563213 step 0 next 1646
    2021-08-09 03:52:38.090419: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfeb800 of size 1792 by op Fill action_count 94195467563215 step 0 next 1692
    2021-08-09 03:52:38.090438: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfebf00 of size 1792 by op Fill action_count 94195467563216 step 0 next 1654
    2021-08-09 03:52:38.090457: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfec600 of size 1792 by op Fill action_count 94195467563217 step 0 next 87
    2021-08-09 03:52:38.090476: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfecd00 of size 1024 by op Fill action_count 94195467563219 step 0 next 2186
    2021-08-09 03:52:38.090495: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfed100 of size 1024 by op Fill action_count 94195467563220 step 0 next 1407
    2021-08-09 03:52:38.090513: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfed500 of size 1024 by op Fill action_count 94195467563221 step 0 next 937
    2021-08-09 03:52:38.090544: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfed900 of size 1024 by op Fill action_count 94195467563223 step 0 next 1812
    2021-08-09 03:52:38.090561: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfedd00 of size 1024 by op Fill action_count 94195467563224 step 0 next 2446
    2021-08-09 03:52:38.090576: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfee100 of size 1024 by op Fill action_count 94195467563225 step 0 next 1989
    2021-08-09 03:52:38.090595: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfee500 of size 1024 by op Fill action_count 94195467563227 step 0 next 2192
    2021-08-09 03:52:38.090614: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfee900 of size 1024 by op Fill action_count 94195467563228 step 0 next 473
    2021-08-09 03:52:38.090635: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfeed00 of size 1024 by op Fill action_count 94195467563229 step 0 next 1201
    2021-08-09 03:52:38.090653: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfef100 of size 1024 by op Fill action_count 94195467563230 step 0 next 2573
    2021-08-09 03:52:38.090672: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfef500 of size 256 by op Fill action_count 94195467563231 step 0 next 508
    2021-08-09 03:52:38.090691: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfef600 of size 1536 by op Fill action_count 94195467563232 step 0 next 684
    2021-08-09 03:52:38.090710: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfefc00 of size 512 by op Fill action_count 94195467563233 step 0 next 579
    2021-08-09 03:52:38.090730: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfefe00 of size 512 by op Fill action_count 94195467563234 step 0 next 1756
    2021-08-09 03:52:38.090749: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff0000 of size 512 by op Fill action_count 94195467563235 step 0 next 1766
    2021-08-09 03:52:38.090768: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff0200 of size 512 by op Fill action_count 94195467563237 step 0 next 648
    2021-08-09 03:52:38.090787: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff0400 of size 512 by op Fill action_count 94195467563238 step 0 next 2663
    2021-08-09 03:52:38.090806: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff0600 of size 512 by op Fill action_count 94195467563239 step 0 next 527
    2021-08-09 03:52:38.090825: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff0800 of size 1024 by op Fill action_count 94195467563241 step 0 next 267
    2021-08-09 03:52:38.090844: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff0c00 of size 1024 by op Fill action_count 94195467563242 step 0 next 553
    2021-08-09 03:52:38.090860: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff1000 of size 1024 by op Fill action_count 94195467563243 step 0 next 1850
    2021-08-09 03:52:38.090880: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff1400 of size 1024 by op Fill action_count 94195467563245 step 0 next 235
    2021-08-09 03:52:38.090899: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff1800 of size 1024 by op Fill action_count 94195467563246 step 0 next 550
    2021-08-09 03:52:38.090918: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff1c00 of size 1024 by op Fill action_count 94195467563247 step 0 next 323
    2021-08-09 03:52:38.090937: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff2000 of size 1792 by op Fill action_count 94195467563249 step 0 next 723
    2021-08-09 03:52:38.090956: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff2700 of size 1792 by op Fill action_count 94195467563250 step 0 next 152
    2021-08-09 03:52:38.090975: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff2e00 of size 2304 by op Fill action_count 94195467563251 step 0 next 1742
    2021-08-09 03:52:38.090992: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff3700 of size 256 by op Sub action_count 94195094149762 step 0 next 1696
    2021-08-09 03:52:38.091008: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff3800 of size 256 by op Sub action_count 94195094149763 step 0 next 1624
    2021-08-09 03:52:38.091027: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff3900 of size 3584 by op Fill action_count 94195455013762 step 0 next 495
    2021-08-09 03:52:38.091047: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff4700 of size 6656 by op Fill action_count 94195455013764 step 0 next 464
    2021-08-09 03:52:38.091063: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff6100 of size 256 by op Sub action_count 94195094149800 step 0 next 1592
    2021-08-09 03:52:38.091079: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff6200 of size 256 by op Sub action_count 94195094149801 step 0 next 1619
    2021-08-09 03:52:38.091095: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff6300 of size 1024 by op Fill action_count 94195455013599 step 0 next 1110
    2021-08-09 03:52:38.091115: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff6700 of size 1024 by op Fill action_count 94195455013600 step 0 next 1927
    2021-08-09 03:52:38.091134: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff6b00 of size 256 by op Fill action_count 94195455013610 step 0 next 2528
    2021-08-09 03:52:38.091153: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff6c00 of size 256 by op Sub action_count 94195480110856 step 0 next 842
    2021-08-09 03:52:38.091175: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff6d00 of size 256 by op AssignAddVariableOp_10 action_count 94195467561440 step 17436438550591945738 next 2050
    2021-08-09 03:52:38.091192: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff6e00 of size 256 by op Sub action_count 94195480110857 step 0 next 1259
    2021-08-09 03:52:38.091208: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff6f00 of size 256 by op Sub action_count 94195455013602 step 0 next 2800
    2021-08-09 03:52:38.091224: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff7000 of size 256 by op Sub action_count 94195455013603 step 0 next 2780
    2021-08-09 03:52:38.091239: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff7100 of size 256 by op AssignVariableOp action_count 94195467505275 step 0 next 44
    2021-08-09 03:52:38.091260: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff7200 of size 256 by op AssignVariableOp action_count 94195467505273 step 0 next 823
    2021-08-09 03:52:38.091279: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff7300 of size 256 by op AssignVariableOp action_count 94195467505277 step 0 next 819
    2021-08-09 03:52:38.091298: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff7400 of size 256 by op Sub action_count 94195480110870 step 0 next 2018
    2021-08-09 03:52:38.091318: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff7500 of size 256 by op Const action_count 94195455640678 step 0 next 3047
    2021-08-09 03:52:38.091339: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff7600 of size 1536 by op Add action_count 94195455013607 step 0 next 1651
    2021-08-09 03:52:38.091355: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff7c00 of size 256 by op Sub action_count 94195094149838 step 0 next 1743
    2021-08-09 03:52:38.091371: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff7d00 of size 256 by op Sub action_count 94195094149839 step 0 next 1559
    2021-08-09 03:52:38.091387: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff7e00 of size 256 by op Sub action_count 94195299052790 step 0 next 409
    2021-08-09 03:52:38.091403: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff7f00 of size 2560 by op Fill action_count 94195455013495 step 0 next 2405
    2021-08-09 03:52:38.091419: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff8900 of size 256 by op Sub action_count 94195299052803 step 0 next 343
    2021-08-09 03:52:38.091435: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff8a00 of size 256 by op Sub action_count 94195455013498 step 0 next 2753
    2021-08-09 03:52:38.091454: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff8b00 of size 256 by op Sub action_count 94195455013499 step 0 next 2925
    2021-08-09 03:52:38.091470: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff8c00 of size 2304 by op Fill action_count 94195455013506 step 0 next 1906
    2021-08-09 03:52:38.091487: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff9500 of size 256 by op Sub action_count 94195299052804 step 0 next 264
    2021-08-09 03:52:38.091503: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff9600 of size 256 by op Sub action_count 94195273335761 step 0 next 2479
    2021-08-09 03:52:38.091527: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff9700 of size 1536 by op Fill action_count 94195455013424 step 0 next 2409
    2021-08-09 03:52:38.091563: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff9d00 of size 256 by op Sub action_count 94195273335762 step 0 next 213
    2021-08-09 03:52:38.091596: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbff9e00 of size 1536 by op Fill action_count 94195455013665 step 0 next 1614
    2021-08-09 03:52:38.091630: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffa400 of size 512 by op Fill action_count 94195455013667 step 0 next 2799
    2021-08-09 03:52:38.091652: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffa600 of size 512 by op Fill action_count 94195455013668 step 0 next 3076
    2021-08-09 03:52:38.091671: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffa800 of size 512 by op Fill action_count 94195455013670 step 0 next 1341
    2021-08-09 03:52:38.091690: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffaa00 of size 512 by op Fill action_count 94195455013671 step 0 next 2899
    2021-08-09 03:52:38.091709: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffac00 of size 512 by op Fill action_count 94195455013672 step 0 next 2602
    2021-08-09 03:52:38.091728: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffae00 of size 1024 by op Fill action_count 94195455013674 step 0 next 2624
    2021-08-09 03:52:38.091747: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffb200 of size 1024 by op Fill action_count 94195455013675 step 0 next 3042
    2021-08-09 03:52:38.091766: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffb600 of size 1280 by op Fill action_count 94195455013676 step 0 next 1868
    2021-08-09 03:52:38.091785: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffbb00 of size 256 by op Sub action_count 94195273335803 step 0 next 536
    2021-08-09 03:52:38.091804: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffbc00 of size 256 by op Sub action_count 94195273335804 step 0 next 1269
    2021-08-09 03:52:38.091823: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffbd00 of size 3840 by op Fill action_count 94195455013467 step 0 next 332
    2021-08-09 03:52:38.091841: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffcc00 of size 256 by op Sub action_count 94195273335775 step 0 next 403
    2021-08-09 03:52:38.091858: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffcd00 of size 256 by op assert_less_equal_5/Assert/AssertGuard/pivot_f/_153 action_count 94195455013856 step 0 next 1672
    2021-08-09 03:52:38.091876: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffce00 of size 256 by op assert_less_equal_5/Assert/AssertGuard/pivot_t/_154 action_count 94195455013857 step 0 next 2153
    2021-08-09 03:52:38.091896: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffcf00 of size 256 by op assert_greater_equal_6/Assert/AssertGuard/pivot_f/_171 action_count 94195455013858 step 0 next 2223
    2021-08-09 03:52:38.091916: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffd000 of size 256 by op assert_greater_equal_6/Assert/AssertGuard/pivot_t/_172 action_count 94195455013859 step 0 next 978
    2021-08-09 03:52:38.091933: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffd100 of size 256 by op assert_less_equal_6/Assert/AssertGuard/pivot_f/_181 action_count 94195455013860 step 0 next 2072
    2021-08-09 03:52:38.091950: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffd200 of size 256 by op assert_less_equal_6/Assert/AssertGuard/pivot_t/_182 action_count 94195455013861 step 0 next 1554
    2021-08-09 03:52:38.091968: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffd300 of size 256 by op gradient_tape/unet_depth3/two_conv_decoder1/batch_normalization_559/moments/scalar action_count 94195455013862
    step 0 next 2185
    2021-08-09 03:52:38.091989: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffd400 of size 256 by op assert_greater_equal_7/Assert/AssertGuard/pivot_f/_199 action_count 94195455013863 step 0 next 2208
    2021-08-09 03:52:38.092011: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffd500 of size 256 by op assert_greater_equal_7/Assert/AssertGuard/pivot_t/_200 action_count 94195455013864 step 0 next 1883
    2021-08-09 03:52:38.092032: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffd600 of size 256 by op assert_less_equal_7/Assert/AssertGuard/pivot_f/_209 action_count 94195455013865 step 0 next 2040
    2021-08-09 03:52:38.092052: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffd700 of size 256 by op assert_less_equal_7/Assert/AssertGuard/pivot_t/_210 action_count 94195455013866 step 0 next 2211
    2021-08-09 03:52:38.092071: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffd800 of size 256 by op assert_greater_equal_8/Assert/AssertGuard/pivot_f/_227 action_count 94195455013867 step 0 next 924
    2021-08-09 03:52:38.092091: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffd900 of size 256 by op assert_greater_equal_8/Assert/AssertGuard/pivot_t/_228 action_count 94195455013868 step 0 next 24
    2021-08-09 03:52:38.092109: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffda00 of size 256 by op gradient_tape/unet_depth3/two_conv_decoder1/batch_normalization_558/moments/scalar action_count 94195455013869
    step 0 next 2162
    2021-08-09 03:52:38.092131: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffdb00 of size 256 by op assert_less_equal_8/Assert/AssertGuard/pivot_f/_237 action_count 94195455013870 step 0 next 2206
    2021-08-09 03:52:38.092148: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffdc00 of size 256 by op assert_less_equal_8/Assert/AssertGuard/pivot_t/_238 action_count 94195455013871 step 0 next 2085
    2021-08-09 03:52:38.092166: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffdd00 of size 256 by op assert_greater_equal_9/Assert/AssertGuard/pivot_f/_255 action_count 94195455013872 step 0 next 1053
    2021-08-09 03:52:38.092188: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffde00 of size 256 by op assert_greater_equal_9/Assert/AssertGuard/pivot_t/_256 action_count 94195455013873 step 0 next 2159
    2021-08-09 03:52:38.092205: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffdf00 of size 256 by op assert_less_equal_9/Assert/AssertGuard/pivot_f/_265 action_count 94195455013874 step 0 next 891
    2021-08-09 03:52:38.092226: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffe000 of size 256 by op assert_less_equal_9/Assert/AssertGuard/pivot_t/_266 action_count 94195455013875 step 0 next 728
    2021-08-09 03:52:38.092245: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffe100 of size 256 by op gradient_tape/unet_depth3/conv_transpose_decoder1/batch_normalization_557/moments/scalar action_count 941954550
    13876 step 0 next 2077
    2021-08-09 03:52:38.092265: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffe200 of size 256 by op assert_greater_equal_10/Assert/AssertGuard/pivot_f/_283 action_count 94195455013877 step 0 next 2230
    2021-08-09 03:52:38.092285: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffe300 of size 256 by op assert_greater_equal_10/Assert/AssertGuard/pivot_t/_284 action_count 94195455013878 step 0 next 2237
    2021-08-09 03:52:38.092307: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffe400 of size 256 by op assert_less_equal_10/Assert/AssertGuard/pivot_f/_293 action_count 94195455013879 step 0 next 2207
    2021-08-09 03:52:38.092329: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffe500 of size 256 by op assert_less_equal_10/Assert/AssertGuard/pivot_t/_294 action_count 94195455013880 step 0 next 1983
    2021-08-09 03:52:38.092346: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffe600 of size 256 by op assert_greater_equal_11/Assert/AssertGuard/pivot_f/_311 action_count 94195455013881 step 0 next 712
    2021-08-09 03:52:38.092364: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffe700 of size 256 by op assert_greater_equal_11/Assert/AssertGuard/pivot_t/_312 action_count 94195455013882 step 0 next 1848
    2021-08-09 03:52:38.092381: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffe800 of size 256 by op gradient_tape/unet_depth3/two_conv_decoder2/batch_normalization_556/moments/scalar action_count 94195455013883
    step 0 next 2205
    2021-08-09 03:52:38.092399: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffe900 of size 256 by op assert_less_equal_11/Assert/AssertGuard/pivot_f/_321 action_count 94195455013884 step 0 next 1795
    2021-08-09 03:52:38.092419: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffea00 of size 256 by op assert_less_equal_11/Assert/AssertGuard/pivot_t/_322 action_count 94195455013885 step 0 next 2227
    2021-08-09 03:52:38.092441: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffeb00 of size 256 by op assert_greater_equal_12/Assert/AssertGuard/pivot_f/_339 action_count 94195455013886 step 0 next 2231
    2021-08-09 03:52:38.092462: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffec00 of size 256 by op assert_greater_equal_12/Assert/AssertGuard/pivot_t/_340 action_count 94195455013887 step 0 next 1767
    2021-08-09 03:52:38.092483: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffed00 of size 256 by op assert_less_equal_12/Assert/AssertGuard/pivot_f/_349 action_count 94195455013888 step 0 next 1785
    2021-08-09 03:52:38.092501: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffee00 of size 256 by op assert_less_equal_12/Assert/AssertGuard/pivot_t/_350 action_count 94195455013889 step 0 next 1151
    2021-08-09 03:52:38.092528: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffef00 of size 256 by op gradient_tape/unet_depth3/two_conv_decoder2/batch_normalization_555/moments/scalar action_count 94195455013890
    step 0 next 2139
    2021-08-09 03:52:38.092562: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfff000 of size 256 by op assert_greater_equal_13/Assert/AssertGuard/pivot_f/_367 action_count 94195455013891 step 0 next 192
    2021-08-09 03:52:38.092595: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfff100 of size 256 by op assert_greater_equal_13/Assert/AssertGuard/pivot_t/_368 action_count 94195455013892 step 0 next 2203
    2021-08-09 03:52:38.092618: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfff200 of size 256 by op assert_less_equal_13/Assert/AssertGuard/pivot_f/_377 action_count 94195455013893 step 0 next 2020
    2021-08-09 03:52:38.092635: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfff300 of size 256 by op assert_less_equal_13/Assert/AssertGuard/pivot_t/_378 action_count 94195455013894 step 0 next 2234
    2021-08-09 03:52:38.092652: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfff400 of size 256 by op assert_greater_equal_14/Assert/AssertGuard/pivot_f/_395 action_count 94195455013895 step 0 next 2428
    2021-08-09 03:52:38.092672: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfff500 of size 256 by op assert_greater_equal_14/Assert/AssertGuard/pivot_t/_396 action_count 94195455013896 step 0 next 2429
    2021-08-09 03:52:38.092692: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfff600 of size 256 by op assert_less_equal_14/Assert/AssertGuard/pivot_f/_405 action_count 94195455013897 step 0 next 2167
    2021-08-09 03:52:38.092712: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfff700 of size 256 by op assert_less_equal_14/Assert/AssertGuard/pivot_t/_406 action_count 94195455013898 step 0 next 186
    2021-08-09 03:52:38.092732: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfff800 of size 256 by op gradient_tape/unet_depth3/conv_transpose_decoder2/batch_normalization_554/moments/scalar action_count 941954550
    13899 step 0 next 1800
    2021-08-09 03:52:38.092752: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfff900 of size 256 by op assert_greater_equal_15/Assert/AssertGuard/pivot_f/_423 action_count 94195455013900 step 0 next 650
    2021-08-09 03:52:38.092772: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfffa00 of size 256 by op assert_greater_equal_15/Assert/AssertGuard/pivot_t/_424 action_count 94195455013901 step 0 next 2488
    2021-08-09 03:52:38.092793: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfffb00 of size 256 by op assert_less_equal_15/Assert/AssertGuard/pivot_f/_433 action_count 94195455013902 step 0 next 2124
    2021-08-09 03:52:38.092815: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfffc00 of size 256 by op assert_less_equal_15/Assert/AssertGuard/pivot_t/_434 action_count 94195455013903 step 0 next 82
    2021-08-09 03:52:38.092838: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfffd00 of size 256 by op assert_greater_equal_16/Assert/AssertGuard/pivot_f/_451 action_count 94195455013904 step 0 next 2047
    2021-08-09 03:52:38.092856: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbfffe00 of size 256 by op assert_greater_equal_16/Assert/AssertGuard/pivot_t/_452 action_count 94195455013905 step 0 next 2183
    2021-08-09 03:52:38.092876: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3abbffff00 of size 256 by op gradient_tape/unet_depth3/two_conv_center/batch_normalization_553/moments/scalar action_count 94195455013906 st
    ep 0 next 18446744073709551615
    2021-08-09 03:52:38.092894: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Next region of size 1073741824
    2021-08-09 03:52:38.092914: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8000000 of size 3328 by op Fill action_count 94195455013519 step 0 next 2009
    2021-08-09 03:52:38.092931: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8000d00 of size 256 by op Sub action_count 94195162895941 step 0 next 934
    2021-08-09 03:52:38.092948: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8000e00 of size 13459968 by op Add action_count 94195455013475 step 0 next 1093
    2021-08-09 03:52:38.092968: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd7000 of size 512 by op Fill action_count 94195410012592 step 0 next 2129
    2021-08-09 03:52:38.092987: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd7200 of size 512 by op Fill action_count 94195410012593 step 0 next 2577
    2021-08-09 03:52:38.093010: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd7400 of size 256 by op AssignAddVariableOp_28 action_count 94195455011291 step 18076379956407770847 next 340
    2021-08-09 03:52:38.093028: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd7500 of size 256 by op AssignAddVariableOp_26 action_count 94195455011239 step 18076379956407770847 next 1402
    2021-08-09 03:52:38.093051: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd7600 of size 256 by op AssignAddVariableOp_36 action_count 94195455011295 step 18076379956407770847 next 2803
    2021-08-09 03:52:38.093069: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd7700 of size 256 by op AssignAddVariableOp_34 action_count 94195455011251 step 18076379956407770847 next 183
    2021-08-09 03:52:38.093089: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd7800 of size 256 by op Fill action_count 94195410012556 step 0 next 758
    2021-08-09 03:52:38.093110: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd7900 of size 256 by op AssignVariableOp action_count 94195454831374 step 0 next 810
    2021-08-09 03:52:38.093130: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd7a00 of size 256 by op Fill action_count 94195410012572 step 0 next 253
    2021-08-09 03:52:38.093149: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd7b00 of size 256 by op Fill action_count 94195410012880 step 0 next 2035
    2021-08-09 03:52:38.093168: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd7c00 of size 256 by op AssignVariableOp action_count 94195454831376 step 0 next 621
    2021-08-09 03:52:38.093188: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd7d00 of size 512 by op Fill action_count 94195410012594 step 0 next 809
    2021-08-09 03:52:38.093207: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd7f00 of size 512 by op Fill action_count 94195410012595 step 0 next 840
    2021-08-09 03:52:38.093226: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd8100 of size 512 by op Fill action_count 94195410012596 step 0 next 1291
    2021-08-09 03:52:38.093242: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd8300 of size 256 by op Sub action_count 94195455013457 step 0 next 1971
    2021-08-09 03:52:38.093258: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd8400 of size 768 by op Fill action_count 94195410012607 step 0 next 1035
    2021-08-09 03:52:38.093274: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd8700 of size 768 by op Fill action_count 94195410012608 step 0 next 2087
    2021-08-09 03:52:38.093290: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd8a00 of size 1024 by op Fill action_count 94195410012616 step 0 next 1381
    2021-08-09 03:52:38.093306: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd8e00 of size 2048 by op Fill action_count 94195467563027 step 0 next 1329
    2021-08-09 03:52:38.093322: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd9600 of size 1792 by op Fill action_count 94195467563035 step 0 next 266
    2021-08-09 03:52:38.093342: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cd9d00 of size 1792 by op Fill action_count 94195467563036 step 0 next 2370
    2021-08-09 03:52:38.093362: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cda400 of size 1792 by op Fill action_count 94195467563037 step 0 next 2210
    2021-08-09 03:52:38.093382: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cdab00 of size 1792 by op Fill action_count 94195467563038 step 0 next 3059
    2021-08-09 03:52:38.093401: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cdb200 of size 1792 by op Fill action_count 94195467563039 step 0 next 595
    2021-08-09 03:52:38.093420: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cdb900 of size 1792 by op Fill action_count 94195467563047 step 0 next 2412
    2021-08-09 03:52:38.093439: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cdc000 of size 1792 by op Fill action_count 94195467563048 step 0 next 2597
    2021-08-09 03:52:38.093459: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cdc700 of size 1792 by op Fill action_count 94195467563049 step 0 next 2039
    2021-08-09 03:52:38.093483: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cdce00 of size 1792 by op Fill action_count 94195467563050 step 0 next 2016
    2021-08-09 03:52:38.093503: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cdd500 of size 1792 by op Fill action_count 94195467563051 step 0 next 2422
    2021-08-09 03:52:38.093526: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cddc00 of size 1792 by op Fill action_count 94195467563059 step 0 next 789
    2021-08-09 03:52:38.093546: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cde300 of size 1792 by op Fill action_count 94195467563060 step 0 next 1337
    2021-08-09 03:52:38.093567: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cdea00 of size 1792 by op Fill action_count 94195467563061 step 0 next 2351
    2021-08-09 03:52:38.093583: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cdf100 of size 1792 by op Fill action_count 94195467563062 step 0 next 1763
    2021-08-09 03:52:38.093603: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cdf800 of size 1792 by op Fill action_count 94195467563063 step 0 next 732
    2021-08-09 03:52:38.093619: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8cdff00 of size 1024 by op Fill action_count 94195467563071 step 0 next 2430
    2021-08-09 03:52:38.093635: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce0300 of size 1024 by op Fill action_count 94195467563072 step 0 next 1969
    2021-08-09 03:52:38.093651: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce0700 of size 1024 by op Fill action_count 94195467563073 step 0 next 1127
    2021-08-09 03:52:38.093671: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce0b00 of size 1024 by op Fill action_count 94195467563074 step 0 next 1752
    2021-08-09 03:52:38.093690: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce0f00 of size 1024 by op Fill action_count 94195467563075 step 0 next 1139
    2021-08-09 03:52:38.093707: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce1300 of size 1024 by op Fill action_count 94195467563083 step 0 next 1652
    2021-08-09 03:52:38.093726: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce1700 of size 768 by op Fill action_count 94195410012582 step 0 next 1429
    2021-08-09 03:52:38.093748: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce1a00 of size 256 by op AssignAddVariableOp_11 action_count 94195455011217 step 18076379956407770847 next 2936
    2021-08-09 03:52:38.093769: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce1b00 of size 256 by op AssignAddVariableOp_20 action_count 94195455011287 step 18076379956407770847 next 1330
    2021-08-09 03:52:38.093787: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce1c00 of size 256 by op AssignAddVariableOp_18 action_count 94195455011227 step 18076379956407770847 next 2503
    2021-08-09 03:52:38.093807: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce1d00 of size 256 by op Fill action_count 94195410012557 step 0 next 1716
    2021-08-09 03:52:38.093827: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce1e00 of size 256 by op AssignVariableOp action_count 94195454831386 step 0 next 1521
    2021-08-09 03:52:38.093843: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce1f00 of size 256 by op Fill action_count 94195410012558 step 0 next 2619
    2021-08-09 03:52:38.093863: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce2000 of size 512 by op Fill action_count 94195455013397 step 0 next 1784
    2021-08-09 03:52:38.093882: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce2200 of size 512 by op Fill action_count 94195455013398 step 0 next 1957
    2021-08-09 03:52:38.093898: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce2400 of size 512 by op Fill action_count 94195410012580 step 0 next 1687
    2021-08-09 03:52:38.093914: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce2600 of size 1536 by op Fill action_count 94195455013425 step 0 next 864
    2021-08-09 03:52:38.093930: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce2c00 of size 768 by op Fill action_count 94195410012617 step 0 next 1205
    2021-08-09 03:52:38.093946: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce2f00 of size 768 by op Fill action_count 94195410012618 step 0 next 2439
    2021-08-09 03:52:38.093962: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce3200 of size 768 by op Fill action_count 94195410012619 step 0 next 328
    2021-08-09 03:52:38.093982: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce3500 of size 768 by op Fill action_count 94195410012620 step 0 next 1959
    2021-08-09 03:52:38.093998: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce3800 of size 2304 by op Fill action_count 94195410012628 step 0 next 806
    2021-08-09 03:52:38.094017: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce4100 of size 256 by op Sub action_count 94195455013470 step 0 next 807
    2021-08-09 03:52:38.094037: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac8ce4200 of size 6291456 by op Add action_count 94195410012673 step 0 next 285
    2021-08-09 03:52:38.094056: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac92e4200 of size 6291456 by op Fill action_count 94195410012975 step 0 next 2275
    2021-08-09 03:52:38.094073: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ac98e4200 of size 18874368 by op Fill action_count 94195410012979 step 0 next 766
    2021-08-09 03:52:38.094092: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acaae4200 of size 12582912 by op Add action_count 94195410012685 step 0 next 2480
    2021-08-09 03:52:38.094109: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acb6e4200 of size 3145728 by op Add action_count 94195410012757 step 0 next 1350
    2021-08-09 03:52:38.094128: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acb9e4200 of size 3145728 by op Fill action_count 94195410012971 step 0 next 1449
    2021-08-09 03:52:38.094148: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acbce4200 of size 6291456 by op Add action_count 94195410012709 step 0 next 1797
    2021-08-09 03:52:38.094164: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acc2e4200 of size 4767744 by op Fill action_count 94195410012983 step 0 next 1516
    2021-08-09 03:52:38.094184: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acc770200 of size 7193600 by op Add action_count 94195410012745 step 0 next 1899
    2021-08-09 03:52:38.094200: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acce4c600 of size 3072 by op Fill action_count 94195455013509 step 0 next 140
    2021-08-09 03:52:38.094217: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acce4d200 of size 256 by op Sub action_count 94195347535784 step 0 next 80
    2021-08-09 03:52:38.094236: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acce4d300 of size 256 by op Sub action_count 94195347535785 step 0 next 2606
    2021-08-09 03:52:38.094252: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acce4d400 of size 256 by op Sub action_count 94195347535798 step 0 next 174
    2021-08-09 03:52:38.094272: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acce4d500 of size 256 by op Sub action_count 94195347535799 step 0 next 1629
    2021-08-09 03:52:38.094291: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acce4d600 of size 256 by op Sub action_count 94195347535824 step 0 next 2022
    2021-08-09 03:52:38.094310: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acce4d700 of size 2198272 by op Add action_count 94195455013555 step 0 next 3068
    2021-08-09 03:52:38.094327: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd066200 of size 824576 by op Add action_count 94195455013593 step 0 next 2543
    2021-08-09 03:52:38.094346: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd12f700 of size 1532928 by op Add action_count 94195455013581 step 0 next 237
    2021-08-09 03:52:38.094366: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd2a5b00 of size 256 by op Sub action_count 94195347535825 step 0 next 342
    2021-08-09 03:52:38.094382: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd2a5c00 of size 2048 by op Fill action_count 94195455013522 step 0 next 2807
    2021-08-09 03:52:38.094401: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd2a6400 of size 256 by op Sub action_count 94195455013524 step 0 next 2157
    2021-08-09 03:52:38.094420: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd2a6500 of size 256 by op Sub action_count 94195455013525 step 0 next 2992
    2021-08-09 03:52:38.094440: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd2a6600 of size 2304 by op Fill action_count 94195455013532 step 0 next 1806
    2021-08-09 03:52:38.094459: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd2a6f00 of size 256 by op Sub action_count 94195365012495 step 0 next 2064
    2021-08-09 03:52:38.094480: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd2a7000 of size 4046080 by op Add action_count 94195455013447 step 0 next 364
    2021-08-09 03:52:38.094497: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd682d00 of size 256 by op Sub action_count 94195347535729 step 0 next 385
    2021-08-09 03:52:38.094512: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd682e00 of size 256 by op Sub action_count 94195347535742 step 0 next 2240
    2021-08-09 03:52:38.094542: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd682f00 of size 1024 by op Fill action_count 94195455013572 step 0 next 3089
    2021-08-09 03:52:38.094563: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd683300 of size 1024 by op Fill action_count 94195455013573 step 0 next 613
    2021-08-09 03:52:38.094582: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd683700 of size 1024 by op Fill action_count 94195455013574 step 0 next 1338
    2021-08-09 03:52:38.094602: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd683b00 of size 256 by op Sub action_count 94195455013576 step 0 next 1873
    2021-08-09 03:52:38.094618: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd683c00 of size 256 by op Sub action_count 94195455013577 step 0 next 792
    2021-08-09 03:52:38.094638: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd683d00 of size 1024 by op Fill action_count 94195455013584 step 0 next 2937
    2021-08-09 03:52:38.094657: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd684100 of size 1280 by op Fill action_count 94195455013585 step 0 next 1544
    2021-08-09 03:52:38.094677: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd684600 of size 256 by op Sub action_count 94195347535743 step 0 next 604
    2021-08-09 03:52:38.094697: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd684700 of size 256 by op Sub action_count 94195347535756 step 0 next 1271
    2021-08-09 03:52:38.094716: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acd684800 of size 14024704 by op Add action_count 94195455013489 step 0 next 2985
    2021-08-09 03:52:38.094733: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ace3e4800 of size 5775360 by op Fill action_count 94195455013701 step 0 next 1295
    2021-08-09 03:52:38.094752: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ace966800 of size 7012352 by op Fill action_count 94195455013709 step 0 next 1346
    2021-08-09 03:52:38.094771: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acf016800 of size 3297536 by op Fill action_count 94195455013713 step 0 next 2856
    2021-08-09 03:52:38.094788: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acf33b900 of size 2198272 by op Fill action_count 94195455013717 step 0 next 1339
    2021-08-09 03:52:38.094806: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acf554400 of size 2930944 by op Fill action_count 94195455013721 step 0 next 3051
    2021-08-09 03:52:38.094825: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acf81fd00 of size 1099264 by op Fill action_count 94195455013751 step 0 next 1099
    2021-08-09 03:52:38.094844: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acf92c300 of size 2198272 by op Fill action_count 94195455013755 step 0 next 2264
    2021-08-09 03:52:38.094864: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acfb44e00 of size 4396544 by op Fill action_count 94195455013759 step 0 next 1717
    2021-08-09 03:52:38.094881: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3acff76400 of size 8792832 by op Fill action_count 94195455013763 step 0 next 1770
    2021-08-09 03:52:38.094901: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad07d8f00 of size 10772736 by op Fill action_count 94195455013771 step 0 next 1790
    2021-08-09 03:52:38.094921: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad121f000 of size 256 by op Sub action_count 94195273335776 step 0 next 825
    2021-08-09 03:52:38.094941: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad121f100 of size 256 by op Sub action_count 94195455013538 step 0 next 566
    2021-08-09 03:52:38.094960: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad121f200 of size 256 by op Sub action_count 94195455013539 step 0 next 3050
    2021-08-09 03:52:38.094979: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad121f300 of size 1792 by op Fill action_count 94195455013546 step 0 next 197
    2021-08-09 03:52:38.094995: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad121fa00 of size 3072 by op Fill action_count 94195455013547 step 0 next 1496
    2021-08-09 03:52:38.095015: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad1220600 of size 256 by op Sub action_count 94195273335789 step 0 next 908
    2021-08-09 03:52:38.095035: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad1220700 of size 256 by op Sub action_count 94195273335790 step 0 next 744
    2021-08-09 03:52:38.095055: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad1220800 of size 256 by op Adam/Pow action_count 94195480111529 step 13755587135625806913 next 3045
    2021-08-09 03:52:38.095094: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad1220900 of size 256 by op Adam/Cast action_count 94195480111527 step 13755587135625806913 next 2731
    2021-08-09 03:52:38.095116: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad1220a00 of size 1441792 by op SameWorkerRecvDone action_count 94195480111530 step 0 next 3056
    2021-08-09 03:52:38.095136: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad1380a00 of size 1441792 by op SameWorkerRecvDone action_count 94195480111531 step 0 next 2916
    2021-08-09 03:52:38.095153: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad14e0a00 of size 1441792 by op binary_ce_dice/mul action_count 94195480111532 step 13755587135625806913 next 2802
    2021-08-09 03:52:38.095174: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad1640a00 of size 360448 by op Cast_4 action_count 94195480111533 step 13755587135625806913 next 2990
    2021-08-09 03:52:38.095191: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad1698a00 of size 360448 by op LogicalNot action_count 94195480111534 step 13755587135625806913 next 2876
    2021-08-09 03:52:38.095209: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad16f0a00 of size 36044800 by op Tile_92 action_count 94195480111535 step 13755587135625806913 next 3077
    2021-08-09 03:52:38.095230: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad3950a00 of size 36044800 by op LogicalNot_31 action_count 94195480111536 step 13755587135625806913 next 2974
    2021-08-09 03:52:38.095247: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad5bb0a00 of size 1441792 by op gradient_tape/binary_ce_dice/logistic_loss/mul/Mul action_count 94195480111537 step 13755587135625806913 ne
    xt 2846
    2021-08-09 03:52:38.095267: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad5d10a00 of size 512 by op unet_depth4/encode0/batch_normalization_580/moments/mean action_count 94195480111555 step 13755587135625806913
    next 3067
    2021-08-09 03:52:38.095288: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad5d10c00 of size 512 by op unet_depth4/encode0/batch_normalization_580/batchnorm/mul action_count 94195480111567 step 13755587135625806913
     next 3027
    2021-08-09 03:52:38.095305: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ad5d10e00 of size 245764096 by op unet_depth4/encode0/batch_normalization_580/batchnorm/mul_1 action_count 94195480111568 step 137555871356
    25806913 next 1011
    2021-08-09 03:52:38.095322: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4771e00 of size 256 by op Sub action_count 94195129464801 step 0 next 1921
    2021-08-09 03:52:38.095340: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4771f00 of size 256 by op Sub action_count 94195129464802 step 0 next 2051
    2021-08-09 03:52:38.095360: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4772000 of size 1024 by op Fill action_count 94195455013678 step 0 next 2527
    2021-08-09 03:52:38.095379: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4772400 of size 1024 by op Fill action_count 94195455013679 step 0 next 2929
    2021-08-09 03:52:38.095398: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4772800 of size 1024 by op Fill action_count 94195455013680 step 0 next 2596
    2021-08-09 03:52:38.095414: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4772c00 of size 1792 by op Fill action_count 94195455013682 step 0 next 1426
    2021-08-09 03:52:38.095434: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4773300 of size 2560 by op Fill action_count 94195455013683 step 0 next 2441
    2021-08-09 03:52:38.095453: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4773d00 of size 256 by op Sub action_count 94195299052817 step 0 next 906
    2021-08-09 03:52:38.095473: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4773e00 of size 256 by op AssignVariableOp action_count 94195454831358 step 0 next 1573
    2021-08-09 03:52:38.095492: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4773f00 of size 256 by op Sub action_count 94195299052818 step 0 next 2171
    2021-08-09 03:52:38.095508: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4774000 of size 2816 by op Fill action_count 94195455013507 step 0 next 2490
    2021-08-09 03:52:38.095531: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4774b00 of size 256 by op Sub action_count 94195299052831 step 0 next 399
    2021-08-09 03:52:38.095550: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4774c00 of size 1280 by op Fill action_count 94195455013411 step 0 next 748
    2021-08-09 03:52:38.095569: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4775100 of size 256 by op Sub action_count 94195299052832 step 0 next 1298
    2021-08-09 03:52:38.095588: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4775200 of size 2304 by op Fill action_count 94195455013493 step 0 next 1658
    2021-08-09 03:52:38.095604: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4775b00 of size 256 by op Sub action_count 94195365012481 step 0 next 505
    2021-08-09 03:52:38.095624: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4775c00 of size 2816 by op Fill action_count 94195455013508 step 0 next 1954
    2021-08-09 03:52:38.095643: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4776700 of size 256 by op Sub action_count 94195129464851 step 0 next 2154
    2021-08-09 03:52:38.095662: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4776800 of size 256 by op Sub action_count 94195129464852 step 0 next 1941
    2021-08-09 03:52:38.095681: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4776900 of size 3584 by op Fill action_count 94195455013690 step 0 next 2822
    2021-08-09 03:52:38.095697: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4777700 of size 4096 by op Fill action_count 94195455013691 step 0 next 2004
    2021-08-09 03:52:38.095717: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4778700 of size 256 by op Sub action_count 94195129464865 step 0 next 2138
    2021-08-09 03:52:38.095735: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4778800 of size 256 by op Sub action_count 94195129464866 step 0 next 1897
    2021-08-09 03:52:38.095755: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4778900 of size 205568 by op Fill action_count 94195467563236 step 0 next 1042
    2021-08-09 03:52:38.095771: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47aac00 of size 256 by op Sub action_count 94195129464999 step 0 next 2123
    2021-08-09 03:52:38.095791: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47aad00 of size 256 by op Sub action_count 94195129465000 step 0 next 1919
    2021-08-09 03:52:38.095810: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47aae00 of size 768 by op Fill action_count 94195480110828 step 0 next 2305
    2021-08-09 03:52:38.095830: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ab100 of size 256 by op AssignAddVariableOp_27 action_count 94195480109387 step 17368542405085625385 next 2028
    2021-08-09 03:52:38.095852: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ab200 of size 256 by op AssignAddVariableOp_29 action_count 94195480109439 step 17368542405085625385 next 740
    2021-08-09 03:52:38.095873: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ab300 of size 256 by op AssignAddVariableOp_35 action_count 94195480109399 step 17368542405085625385 next 324
    2021-08-09 03:52:38.095890: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ab400 of size 512 by op Fill action_count 94195480110826 step 0 next 2241
    2021-08-09 03:52:38.095911: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ab600 of size 256 by op AssignAddVariableOp_37 action_count 94195480109443 step 17368542405085625385 next 892
    2021-08-09 03:52:38.095928: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ab700 of size 256 by op AssignAddVariableOp_4 action_count 94195480109414 step 17368542405085625385 next 413
    2021-08-09 03:52:38.095950: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ab800 of size 256 by op AssignAddVariableOp_2 action_count 94195480109331 step 17368542405085625385 next 1324
    2021-08-09 03:52:38.095966: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ab900 of size 512 by op Fill action_count 94195480110827 step 0 next 957
    2021-08-09 03:52:38.095990: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47abb00 of size 256 by op AssignAddVariableOp_5 action_count 94195480109416 step 17368542405085625385 next 1237
    2021-08-09 03:52:38.096009: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47abc00 of size 256 by op Sub action_count 94195480110871 step 0 next 917
    2021-08-09 03:52:38.096029: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47abd00 of size 256 by op AssignAddVariableOp_26 action_count 94195480109385 step 17368542405085625385 next 2161
    2021-08-09 03:52:38.096045: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47abe00 of size 256 by op AssignAddVariableOp_36 action_count 94195480109441 step 17368542405085625385 next 1557
    2021-08-09 03:52:38.096066: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47abf00 of size 256 by op AssignAddVariableOp_34 action_count 94195480109397 step 17368542405085625385 next 2251
    2021-08-09 03:52:38.096083: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ac000 of size 1024 by op Fill action_count 94195480110865 step 0 next 181
    2021-08-09 03:52:38.096099: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ac400 of size 1024 by op Fill action_count 94195480110866 step 0 next 1358
    2021-08-09 03:52:38.096119: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ac800 of size 1024 by op Fill action_count 94195480110867 step 0 next 1461
    2021-08-09 03:52:38.096138: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47acc00 of size 1024 by op Fill action_count 94195480110868 step 0 next 2290
    2021-08-09 03:52:38.096154: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ad000 of size 2048 by op Fill action_count 94195480110878 step 0 next 68
    2021-08-09 03:52:38.096170: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ad800 of size 2048 by op Fill action_count 94195480110879 step 0 next 314
    2021-08-09 03:52:38.096189: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ae000 of size 2048 by op Fill action_count 94195480110880 step 0 next 1031
    2021-08-09 03:52:38.096207: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ae800 of size 3072 by op Fill action_count 94195480110881 step 0 next 2388
    2021-08-09 03:52:38.096227: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47af400 of size 256 by op AssignVariableOp action_count 94195480053214 step 0 next 256
    2021-08-09 03:52:38.096243: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47af500 of size 512 by op Fill action_count 94195480110838 step 0 next 1188
    2021-08-09 03:52:38.096262: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47af700 of size 512 by op Fill action_count 94195480110839 step 0 next 123
    2021-08-09 03:52:38.096282: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47af900 of size 512 by op Fill action_count 94195480110840 step 0 next 1211
    2021-08-09 03:52:38.096301: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47afb00 of size 1536 by op Add action_count 94195480110821 step 0 next 848
    2021-08-09 03:52:38.096320: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b0100 of size 1024 by op Fill action_count 94195480110852 step 0 next 414
    2021-08-09 03:52:38.096339: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b0500 of size 1024 by op Fill action_count 94195480110853 step 0 next 2495
    2021-08-09 03:52:38.096355: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b0900 of size 1024 by op Fill action_count 94195480110854 step 0 next 1517
    2021-08-09 03:52:38.096374: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b0d00 of size 1024 by op Fill action_count 94195480110864 step 0 next 1878
    2021-08-09 03:52:38.096393: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b1100 of size 256 by op AssignVariableOp action_count 94195480053222 step 0 next 743
    2021-08-09 03:52:38.096409: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b1200 of size 256 by op AssignVariableOp action_count 94195480053224 step 0 next 637
    2021-08-09 03:52:38.096428: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b1300 of size 256 by op AssignVariableOp action_count 94195480053228 step 0 next 261
    2021-08-09 03:52:38.096445: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b1400 of size 2048 by op Fill action_count 94195480110882 step 0 next 667
    2021-08-09 03:52:38.096464: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b1c00 of size 2048 by op Fill action_count 94195480110890 step 0 next 796
    2021-08-09 03:52:38.096483: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b2400 of size 2048 by op Fill action_count 94195480110891 step 0 next 262
    2021-08-09 03:52:38.096499: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b2c00 of size 2048 by op Fill action_count 94195480110892 step 0 next 383
    2021-08-09 03:52:38.096525: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b3400 of size 2048 by op Fill action_count 94195480110893 step 0 next 1383
    2021-08-09 03:52:38.096560: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b3c00 of size 2048 by op Fill action_count 94195480110894 step 0 next 249
    2021-08-09 03:52:38.096592: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b4400 of size 2048 by op Fill action_count 94195480110904 step 0 next 26
    2021-08-09 03:52:38.096612: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b4c00 of size 2048 by op Fill action_count 94195480110905 step 0 next 421
    2021-08-09 03:52:38.096632: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b5400 of size 2048 by op Fill action_count 94195480110906 step 0 next 1905
    2021-08-09 03:52:38.096648: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b5c00 of size 2816 by op Fill action_count 94195480110907 step 0 next 568
    2021-08-09 03:52:38.096671: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b6700 of size 256 by op binary_ce_dice/weighted_loss/num_elements/Cast action_count 94195468133245 step 0 next 896
    2021-08-09 03:52:38.096690: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b6800 of size 512 by op Reshape_92 action_count 94195468133247 step 0 next 1389
    2021-08-09 03:52:38.096709: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b6a00 of size 256 by op Cast_181/x action_count 94195468133253 step 0 next 2512
    2021-08-09 03:52:38.096726: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b6b00 of size 256 by op unet_depth3/encode0/batch_normalization_563/batchnorm/add/y action_count 94195468133254 step 0 next 1801
    2021-08-09 03:52:38.096748: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b6c00 of size 256 by op Cast_180/x action_count 94195468133255 step 0 next 1841
    2021-08-09 03:52:38.096770: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b6d00 of size 256 by op assert_greater_equal/Assert/AssertGuard/pivot_f/_3 action_count 94195468133256 step 0 next 1181
    2021-08-09 03:52:38.096791: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b6e00 of size 256 by op assert_greater_equal/Assert/AssertGuard/pivot_t/_4 action_count 94195468133257 step 0 next 368
    2021-08-09 03:52:38.096813: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b6f00 of size 256 by op Const_35 action_count 94195468133258 step 0 next 1382
    2021-08-09 03:52:38.096834: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7000 of size 256 by op assert_less_equal/Assert/AssertGuard/pivot_f/_13 action_count 94195468133259 step 0 next 1314
    2021-08-09 03:52:38.096851: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7100 of size 256 by op assert_less_equal/Assert/AssertGuard/pivot_t/_14 action_count 94195468133260 step 0 next 1702
    2021-08-09 03:52:38.096872: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7200 of size 256 by op assert_greater_equal_1/Assert/AssertGuard/pivot_f/_31 action_count 94195468133261 step 0 next 180
    2021-08-09 03:52:38.096892: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7300 of size 256 by op assert_greater_equal_1/Assert/AssertGuard/pivot_t/_32 action_count 94195468133262 step 0 next 2133
    2021-08-09 03:52:38.096912: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7400 of size 256 by op assert_less_equal_1/Assert/AssertGuard/pivot_f/_41 action_count 94195468133263 step 0 next 15
    2021-08-09 03:52:38.096931: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7500 of size 256 by op assert_less_equal_1/Assert/AssertGuard/pivot_t/_42 action_count 94195468133264 step 0 next 434
    2021-08-09 03:52:38.096948: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7600 of size 256 by op assert_greater_equal_2/Assert/AssertGuard/pivot_f/_59 action_count 94195468133265 step 0 next 1844
    2021-08-09 03:52:38.096969: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7700 of size 256 by op assert_greater_equal_2/Assert/AssertGuard/pivot_t/_60 action_count 94195468133266 step 0 next 310
    2021-08-09 03:52:38.096987: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7800 of size 256 by op assert_less_equal_2/Assert/AssertGuard/pivot_f/_69 action_count 94195468133267 step 0 next 352
    2021-08-09 03:52:38.097008: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7900 of size 256 by op assert_less_equal_2/Assert/AssertGuard/pivot_t/_70 action_count 94195468133268 step 0 next 1970
    2021-08-09 03:52:38.097029: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7a00 of size 256 by op assert_greater_equal_3/Assert/AssertGuard/pivot_f/_87 action_count 94195468133269 step 0 next 894
    2021-08-09 03:52:38.097046: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7b00 of size 256 by op assert_greater_equal_3/Assert/AssertGuard/pivot_t/_88 action_count 94195468133270 step 0 next 30
    2021-08-09 03:52:38.097066: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7c00 of size 256 by op assert_less_equal_3/Assert/AssertGuard/pivot_f/_97 action_count 94195468133271 step 0 next 1882
    2021-08-09 03:52:38.097083: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7d00 of size 256 by op assert_less_equal_3/Assert/AssertGuard/pivot_t/_98 action_count 94195468133272 step 0 next 300
    2021-08-09 03:52:38.097101: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7e00 of size 256 by op assert_greater_equal_4/Assert/AssertGuard/pivot_f/_115 action_count 94195468133273 step 0 next 936
    2021-08-09 03:52:38.097123: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b7f00 of size 256 by op assert_greater_equal_4/Assert/AssertGuard/pivot_t/_116 action_count 94195468133274 step 0 next 2197
    2021-08-09 03:52:38.097144: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b8000 of size 256 by op assert_less_equal_4/Assert/AssertGuard/pivot_f/_125 action_count 94195468133275 step 0 next 1837
    2021-08-09 03:52:38.097162: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b8100 of size 256 by op assert_less_equal_4/Assert/AssertGuard/pivot_t/_126 action_count 94195468133276 step 0 next 1143
    2021-08-09 03:52:38.097182: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b8200 of size 256 by op AssignAddVariableOp_52 action_count 94195467564196 step 10280291020125615045 next 998
    2021-08-09 03:52:38.097202: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b8300 of size 256 by op Sub action_count 94195480110896 step 0 next 645
    2021-08-09 03:52:38.097220: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b8400 of size 256 by op Sub action_count 94195339127952 step 0 next 1973
    2021-08-09 03:52:38.097240: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b8500 of size 3584 by op Fill action_count 94195455013464 step 0 next 1432
    2021-08-09 03:52:38.097259: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b9300 of size 256 by op Sub action_count 94195339127981 step 0 next 178
    2021-08-09 03:52:38.097275: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b9400 of size 256 by op Sub action_count 94195339127994 step 0 next 925
    2021-08-09 03:52:38.097294: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b9500 of size 256 by op Sub action_count 94195455013471 step 0 next 620
    2021-08-09 03:52:38.097313: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b9600 of size 256 by op AssignVariableOp action_count 94195454831394 step 0 next 1634
    2021-08-09 03:52:38.097329: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b9700 of size 256 by op Sub action_count 94195339127953 step 0 next 1683
    2021-08-09 03:52:38.097345: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b9800 of size 256 by op Sub action_count 94195455013484 step 0 next 2419
    2021-08-09 03:52:38.097361: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b9900 of size 256 by op Sub action_count 94195339128020 step 0 next 2170
    2021-08-09 03:52:38.097376: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b9a00 of size 256 by op Sub action_count 94195339128021 step 0 next 210
    2021-08-09 03:52:38.097396: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b9b00 of size 256 by op Sub action_count 94195365012496 step 0 next 1286
    2021-08-09 03:52:38.097416: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47b9c00 of size 1536 by op Fill action_count 94195455013426 step 0 next 45
    2021-08-09 03:52:38.097432: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ba200 of size 256 by op Sub action_count 94195365012569 step 0 next 2105
    2021-08-09 03:52:38.097448: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ba300 of size 256 by op Sub action_count 94195365012570 step 0 next 1869
    2021-08-09 03:52:38.097467: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ba400 of size 2048 by op Fill action_count 94195455013707 step 0 next 3084
    2021-08-09 03:52:38.097486: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bac00 of size 2048 by op Fill action_count 94195455013708 step 0 next 2958
    2021-08-09 03:52:38.097502: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bb400 of size 1792 by op Fill action_count 94195455013710 step 0 next 2896
    2021-08-09 03:52:38.097526: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bbb00 of size 2304 by op Fill action_count 94195455013711 step 0 next 547
    2021-08-09 03:52:38.097542: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bc400 of size 256 by op Sub action_count 94195365012482 step 0 next 2659
    2021-08-09 03:52:38.097561: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bc500 of size 256 by op Sub action_count 94195455013485 step 0 next 2155
    2021-08-09 03:52:38.097580: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bc600 of size 256 by op Sub action_count 94195129465085 step 0 next 1069
    2021-08-09 03:52:38.097599: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bc700 of size 256 by op Sub action_count 94195129465086 step 0 next 2037
    2021-08-09 03:52:38.097618: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bc800 of size 256 by op AssignVariableOp action_count 94195467505299 step 0 next 2374
    2021-08-09 03:52:38.097638: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bc900 of size 256 by op AssignVariableOp action_count 94195467505301 step 0 next 1012
    2021-08-09 03:52:38.097654: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bca00 of size 256 by op AssignVariableOp action_count 94195467505303 step 0 next 847
    2021-08-09 03:52:38.097673: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bcb00 of size 256 by op Sub action_count 94195480110897 step 0 next 1660
    2021-08-09 03:52:38.097693: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bcc00 of size 256 by op AssignVariableOp action_count 94195467505329 step 0 next 1388
    2021-08-09 03:52:38.097712: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bcd00 of size 256 by op AssignVariableOp action_count 94195467505331 step 0 next 2156
    2021-08-09 03:52:38.097728: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bce00 of size 1024 by op Fill action_count 94195467562927 step 0 next 54
    2021-08-09 03:52:38.097744: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bd200 of size 256 by op Sum_1 action_count 94195455584270 step 0 next 1890
    2021-08-09 03:52:38.097760: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bd300 of size 256 by op truediv/y action_count 94195455584272 step 0 next 511
    2021-08-09 03:52:38.097782: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bd400 of size 256 by op binary_ce_dice/add_1/y action_count 94195455584273 step 0 next 1136
    2021-08-09 03:52:38.097800: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bd500 of size 256 by op binary_ce_dice/weighted_loss/num_elements/Cast action_count 94195455584274 step 0 next 1994
    2021-08-09 03:52:38.097820: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bd600 of size 1792 by op Fill action_count 94195467562951 step 0 next 1932
    2021-08-09 03:52:38.097840: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bdd00 of size 1792 by op Fill action_count 94195467562952 step 0 next 2316
    2021-08-09 03:52:38.097859: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47be400 of size 1792 by op Fill action_count 94195467562953 step 0 next 2172
    2021-08-09 03:52:38.097875: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47beb00 of size 1792 by op Fill action_count 94195467562954 step 0 next 2921
    2021-08-09 03:52:38.097893: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bf200 of size 1792 by op Fill action_count 94195467562955 step 0 next 2489
    2021-08-09 03:52:38.097913: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47bf900 of size 2048 by op Fill action_count 94195467562963 step 0 next 2348
    2021-08-09 03:52:38.097933: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0100 of size 256 by op assert_less_equal_2/Assert/AssertGuard/pivot_f/_69 action_count 94195455584296 step 0 next 1946
    2021-08-09 03:52:38.097950: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0200 of size 256 by op assert_less_equal_2/Assert/AssertGuard/pivot_t/_70 action_count 94195455584297 step 0 next 1886
    2021-08-09 03:52:38.097969: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0300 of size 256 by op assert_greater_equal_3/Assert/AssertGuard/pivot_f/_87 action_count 94195455584298 step 0 next 1458
    2021-08-09 03:52:38.097988: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0400 of size 256 by op assert_greater_equal_3/Assert/AssertGuard/pivot_t/_88 action_count 94195455584299 step 0 next 2400
    2021-08-09 03:52:38.098007: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0500 of size 256 by op assert_less_equal_3/Assert/AssertGuard/pivot_f/_97 action_count 94195455584300 step 0 next 1070
    2021-08-09 03:52:38.098024: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0600 of size 256 by op assert_less_equal_3/Assert/AssertGuard/pivot_t/_98 action_count 94195455584301 step 0 next 2291
    2021-08-09 03:52:38.098043: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0700 of size 256 by op assert_greater_equal_4/Assert/AssertGuard/pivot_f/_115 action_count 94195455584302 step 0 next 1394
    2021-08-09 03:52:38.098060: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0800 of size 256 by op assert_greater_equal_4/Assert/AssertGuard/pivot_t/_116 action_count 94195455584303 step 0 next 2324
    2021-08-09 03:52:38.098079: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0900 of size 256 by op assert_less_equal_4/Assert/AssertGuard/pivot_f/_125 action_count 94195455584304 step 0 next 2372
    2021-08-09 03:52:38.098098: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0a00 of size 256 by op assert_less_equal_4/Assert/AssertGuard/pivot_t/_126 action_count 94195455584305 step 0 next 2271
    2021-08-09 03:52:38.098114: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0b00 of size 256 by op assert_greater_equal_5/Assert/AssertGuard/pivot_f/_143 action_count 94195455584306 step 0 next 1843
    2021-08-09 03:52:38.098134: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0c00 of size 256 by op assert_greater_equal_5/Assert/AssertGuard/pivot_t/_144 action_count 94195455584307 step 0 next 2416
    2021-08-09 03:52:38.098155: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0d00 of size 256 by op assert_less_equal_5/Assert/AssertGuard/pivot_f/_153 action_count 94195455584308 step 0 next 2407
    2021-08-09 03:52:38.098175: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0e00 of size 256 by op assert_less_equal_5/Assert/AssertGuard/pivot_t/_154 action_count 94195455584309 step 0 next 1783
    2021-08-09 03:52:38.098197: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c0f00 of size 256 by op assert_greater_equal_6/Assert/AssertGuard/pivot_f/_171 action_count 94195455584310 step 0 next 2274
    2021-08-09 03:52:38.098218: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c1000 of size 256 by op assert_greater_equal_6/Assert/AssertGuard/pivot_t/_172 action_count 94195455584311 step 0 next 2379
    2021-08-09 03:52:38.098236: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c1100 of size 256 by op assert_less_equal_6/Assert/AssertGuard/pivot_f/_181 action_count 94195455584312 step 0 next 2435
    2021-08-09 03:52:38.098255: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c1200 of size 256 by op assert_less_equal_6/Assert/AssertGuard/pivot_t/_182 action_count 94195455584313 step 0 next 2280
    2021-08-09 03:52:38.098272: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c1300 of size 256 by op assert_greater_equal_7/Assert/AssertGuard/pivot_f/_199 action_count 94195455584314 step 0 next 2447
    2021-08-09 03:52:38.098293: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c1400 of size 256 by op assert_greater_equal_7/Assert/AssertGuard/pivot_t/_200 action_count 94195455584315 step 0 next 1721
    2021-08-09 03:52:38.098314: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c1500 of size 256 by op assert_less_equal_7/Assert/AssertGuard/pivot_f/_209 action_count 94195455584316 step 0 next 1588
    2021-08-09 03:52:38.098335: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c1600 of size 256 by op assert_less_equal_7/Assert/AssertGuard/pivot_t/_210 action_count 94195455584317 step 0 next 2259
    2021-08-09 03:52:38.098351: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c1700 of size 256 by op assert_greater_equal_8/Assert/AssertGuard/pivot_f/_227 action_count 94195455584318 step 0 next 1805
    2021-08-09 03:52:38.098367: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c1800 of size 256 by op Sub action_count 94195129465111 step 0 next 2174
    2021-08-09 03:52:38.098386: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c1900 of size 256 by op Sub action_count 94195129465112 step 0 next 2091
    2021-08-09 03:52:38.098406: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c1a00 of size 1024 by op Fill action_count 94195480111064 step 0 next 1925
    2021-08-09 03:52:38.098422: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c1e00 of size 1024 by op Fill action_count 94195480111065 step 0 next 1834
    2021-08-09 03:52:38.098442: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c2200 of size 1024 by op Fill action_count 94195480111066 step 0 next 1061
    2021-08-09 03:52:38.098460: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c2600 of size 1024 by op Fill action_count 94195480111067 step 0 next 1048
    2021-08-09 03:52:38.098476: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c2a00 of size 1024 by op Fill action_count 94195480111068 step 0 next 284
    2021-08-09 03:52:38.098495: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c2e00 of size 1024 by op Fill action_count 94195480111078 step 0 next 2346
    2021-08-09 03:52:38.098515: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c3200 of size 1024 by op Fill action_count 94195480111079 step 0 next 2238
    2021-08-09 03:52:38.098545: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c3600 of size 1024 by op Fill action_count 94195480111080 step 0 next 1526
    2021-08-09 03:52:38.098565: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c3a00 of size 1024 by op Fill action_count 94195480111081 step 0 next 1710
    2021-08-09 03:52:38.098584: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c3e00 of size 1024 by op Fill action_count 94195480111082 step 0 next 644
    2021-08-09 03:52:38.098603: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c4200 of size 1024 by op Fill action_count 94195480111090 step 0 next 1791
    2021-08-09 03:52:38.098619: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c4600 of size 1024 by op Fill action_count 94195480111091 step 0 next 1623
    2021-08-09 03:52:38.098640: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c4a00 of size 1024 by op Fill action_count 94195480111092 step 0 next 2074
    2021-08-09 03:52:38.098658: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c4e00 of size 1024 by op Fill action_count 94195480111093 step 0 next 2059
    2021-08-09 03:52:38.098677: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c5200 of size 1024 by op Fill action_count 94195480111094 step 0 next 2095
    2021-08-09 03:52:38.098694: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c5600 of size 256 by op Fill action_count 94195480111104 step 0 next 544
    2021-08-09 03:52:38.098713: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c5700 of size 256 by op Fill action_count 94195480111105 step 0 next 1572
    2021-08-09 03:52:38.098732: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c5800 of size 256 by op Fill action_count 94195480111106 step 0 next 18
    2021-08-09 03:52:38.098751: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c5900 of size 256 by op Fill action_count 94195480111107 step 0 next 578
    2021-08-09 03:52:38.098767: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c5a00 of size 256 by op Fill action_count 94195480111108 step 0 next 1311
    2021-08-09 03:52:38.098782: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c5b00 of size 256 by op Fill action_count 94195480111109 step 0 next 1385
    2021-08-09 03:52:38.098801: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c5c00 of size 256 by op Fill action_count 94195480111110 step 0 next 1664
    2021-08-09 03:52:38.098820: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c5d00 of size 256 by op Fill action_count 94195480111111 step 0 next 596
    2021-08-09 03:52:38.098852: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c5e00 of size 1024 by op Add action_count 94195480111099 step 0 next 2190
    2021-08-09 03:52:38.098872: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c6200 of size 256 by op Fill action_count 94195480111112 step 0 next 2621
    2021-08-09 03:52:38.098890: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c6300 of size 256 by op Fill action_count 94195480111113 step 0 next 2146
    2021-08-09 03:52:38.098909: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c6400 of size 256 by op Fill action_count 94195480111114 step 0 next 2195
    2021-08-09 03:52:38.098927: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c6500 of size 256 by op Fill action_count 94195480111115 step 0 next 1111
    2021-08-09 03:52:38.098945: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c6600 of size 256 by op Fill action_count 94195480111116 step 0 next 1789
    2021-08-09 03:52:38.098963: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c6700 of size 256 by op Fill action_count 94195480111117 step 0 next 1301
    2021-08-09 03:52:38.098981: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c6800 of size 256 by op Fill action_count 94195480111118 step 0 next 2286
    2021-08-09 03:52:38.098998: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c6900 of size 256 by op Fill action_count 94195480111119 step 0 next 1706
    2021-08-09 03:52:38.099016: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c6a00 of size 256 by op Fill action_count 94195480111120 step 0 next 858
    2021-08-09 03:52:38.099033: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c6b00 of size 256 by op Fill action_count 94195480111121 step 0 next 2147
    2021-08-09 03:52:38.099048: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c6c00 of size 256 by op Fill action_count 94195480111122 step 0 next 2592
    2021-08-09 03:52:38.099063: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c6d00 of size 256 by op Fill action_count 94195480111123 step 0 next 826
    2021-08-09 03:52:38.099078: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c6e00 of size 256 by op Fill action_count 94195480111124 step 0 next 449
    2021-08-09 03:52:38.099093: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c6f00 of size 256 by op Fill action_count 94195480111125 step 0 next 767
    2021-08-09 03:52:38.099107: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7000 of size 256 by op Fill action_count 94195480111126 step 0 next 347
    2021-08-09 03:52:38.099122: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7100 of size 256 by op Fill action_count 94195480111127 step 0 next 524
    2021-08-09 03:52:38.099140: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7200 of size 256 by op Fill action_count 94195480111128 step 0 next 653
    2021-08-09 03:52:38.099158: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7300 of size 256 by op Fill action_count 94195480111129 step 0 next 764
    2021-08-09 03:52:38.099175: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7400 of size 256 by op Fill action_count 94195480111130 step 0 next 938
    2021-08-09 03:52:38.099193: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7500 of size 256 by op Fill action_count 94195480111131 step 0 next 999
    2021-08-09 03:52:38.099211: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7600 of size 256 by op Fill action_count 94195480111132 step 0 next 916
    2021-08-09 03:52:38.099229: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7700 of size 256 by op Fill action_count 94195480111133 step 0 next 2140
    2021-08-09 03:52:38.099246: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7800 of size 256 by op Fill action_count 94195480111134 step 0 next 523
    2021-08-09 03:52:38.099264: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7900 of size 256 by op Fill action_count 94195480111135 step 0 next 1146
    2021-08-09 03:52:38.099282: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7a00 of size 256 by op Fill action_count 94195480111136 step 0 next 952
    2021-08-09 03:52:38.099300: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7b00 of size 256 by op Fill action_count 94195480111137 step 0 next 423
    2021-08-09 03:52:38.099318: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7c00 of size 256 by op Fill action_count 94195480111138 step 0 next 2423
    2021-08-09 03:52:38.099336: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7d00 of size 256 by op Fill action_count 94195480111139 step 0 next 250
    2021-08-09 03:52:38.099354: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7e00 of size 256 by op Fill action_count 94195480111140 step 0 next 859
    2021-08-09 03:52:38.099373: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c7f00 of size 256 by op Fill action_count 94195480111141 step 0 next 1186
    2021-08-09 03:52:38.099391: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c8000 of size 256 by op Fill action_count 94195480111142 step 0 next 1299
    2021-08-09 03:52:38.099408: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c8100 of size 256 by op Fill action_count 94195480111143 step 0 next 2622
    2021-08-09 03:52:38.099427: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c8200 of size 256 by op Fill action_count 94195480111144 step 0 next 2076
    2021-08-09 03:52:38.099446: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c8300 of size 512 by op Fill action_count 94195480111145 step 0 next 671
    2021-08-09 03:52:38.099461: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c8500 of size 512 by op Fill action_count 94195480111146 step 0 next 1923
    2021-08-09 03:52:38.099479: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c8700 of size 512 by op Fill action_count 94195480111147 step 0 next 2044
    2021-08-09 03:52:38.099497: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c8900 of size 512 by op Fill action_count 94195480111148 step 0 next 1859
    2021-08-09 03:52:38.099515: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c8b00 of size 256 by op Fill action_count 94195480111149 step 0 next 603
    2021-08-09 03:52:38.099535: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c8c00 of size 256 by op Fill action_count 94195480111150 step 0 next 447
    2021-08-09 03:52:38.099550: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c8d00 of size 256 by op Fill action_count 94195480111151 step 0 next 362
    2021-08-09 03:52:38.099565: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c8e00 of size 256 by op Fill action_count 94195480111152 step 0 next 2367
    2021-08-09 03:52:38.099583: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c8f00 of size 256 by op Fill action_count 94195480111153 step 0 next 467
    2021-08-09 03:52:38.099601: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c9000 of size 256 by op Fill action_count 94195480111154 step 0 next 1100
    2021-08-09 03:52:38.099619: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c9100 of size 256 by op AssignVariableOp action_count 94195480111155 step 0 next 91
    2021-08-09 03:52:38.099637: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c9200 of size 256 by op Fill action_count 94195480111156 step 0 next 236
    2021-08-09 03:52:38.099655: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c9300 of size 1536 by op Fill action_count 94195480111157 step 0 next 857
    2021-08-09 03:52:38.099673: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c9900 of size 512 by op Fill action_count 94195480111158 step 0 next 1779
    2021-08-09 03:52:38.099691: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c9b00 of size 512 by op Fill action_count 94195480111159 step 0 next 879
    2021-08-09 03:52:38.099708: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c9d00 of size 512 by op Fill action_count 94195480111160 step 0 next 1191
    2021-08-09 03:52:38.099723: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47c9f00 of size 512 by op Fill action_count 94195480111162 step 0 next 2373
    2021-08-09 03:52:38.099742: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ca100 of size 512 by op Fill action_count 94195480111163 step 0 next 212
    2021-08-09 03:52:38.099760: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ca300 of size 512 by op Fill action_count 94195480111164 step 0 next 27
    2021-08-09 03:52:38.099778: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ca500 of size 1024 by op Fill action_count 94195480111166 step 0 next 2453
    2021-08-09 03:52:38.099796: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ca900 of size 1024 by op Fill action_count 94195480111167 step 0 next 120
    2021-08-09 03:52:38.099814: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47cad00 of size 1024 by op Fill action_count 94195480111168 step 0 next 398
    2021-08-09 03:52:38.099833: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47cb100 of size 1024 by op Fill action_count 94195480111170 step 0 next 394
    2021-08-09 03:52:38.099851: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47cb500 of size 1024 by op Fill action_count 94195480111171 step 0 next 1774
    2021-08-09 03:52:38.099869: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47cb900 of size 1024 by op Fill action_count 94195480111172 step 0 next 881
    2021-08-09 03:52:38.099887: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47cbd00 of size 2048 by op Fill action_count 94195480111174 step 0 next 1019
    2021-08-09 03:52:38.099905: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47cc500 of size 2048 by op Fill action_count 94195480111175 step 0 next 1787
    2021-08-09 03:52:38.099923: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ccd00 of size 2048 by op Fill action_count 94195480111176 step 0 next 768
    2021-08-09 03:52:38.099941: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47cd500 of size 2048 by op Fill action_count 94195480111178 step 0 next 756
    2021-08-09 03:52:38.099959: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47cdd00 of size 2048 by op Fill action_count 94195480111179 step 0 next 1265
    2021-08-09 03:52:38.099978: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ce500 of size 2048 by op Fill action_count 94195480111180 step 0 next 724
    2021-08-09 03:52:38.099997: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ced00 of size 2048 by op Fill action_count 94195480111182 step 0 next 2571
    2021-08-09 03:52:38.100015: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47cf500 of size 2048 by op Fill action_count 94195480111183 step 0 next 562
    2021-08-09 03:52:38.100033: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47cfd00 of size 2048 by op Fill action_count 94195480111184 step 0 next 1038
    2021-08-09 03:52:38.100051: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47d0500 of size 2048 by op Fill action_count 94195480111186 step 0 next 145
    2021-08-09 03:52:38.100069: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47d0d00 of size 2048 by op Fill action_count 94195480111187 step 0 next 1627
    2021-08-09 03:52:38.100086: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47d1500 of size 2048 by op Fill action_count 94195480111188 step 0 next 84
    2021-08-09 03:52:38.100101: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47d1d00 of size 4096 by op Fill action_count 94195480111190 step 0 next 882
    2021-08-09 03:52:38.100116: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47d2d00 of size 4096 by op Fill action_count 94195480111191 step 0 next 529
    2021-08-09 03:52:38.100131: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47d3d00 of size 4096 by op Fill action_count 94195480111192 step 0 next 944
    2021-08-09 03:52:38.100146: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47d4d00 of size 4096 by op Fill action_count 94195480111194 step 0 next 1955
    2021-08-09 03:52:38.100161: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47d5d00 of size 4096 by op Fill action_count 94195480111195 step 0 next 122
    2021-08-09 03:52:38.100176: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47d6d00 of size 4096 by op Fill action_count 94195480111196 step 0 next 1974
    2021-08-09 03:52:38.100194: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47d7d00 of size 2048 by op Fill action_count 94195480111198 step 0 next 1647
    2021-08-09 03:52:38.100212: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47d8500 of size 2048 by op Fill action_count 94195480111199 step 0 next 1830
    2021-08-09 03:52:38.100230: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47d8d00 of size 2048 by op Fill action_count 94195480111200 step 0 next 1244
    2021-08-09 03:52:38.100248: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47d9500 of size 2048 by op Fill action_count 94195480111202 step 0 next 967
    2021-08-09 03:52:38.100266: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47d9d00 of size 2048 by op Fill action_count 94195480111203 step 0 next 1488
    2021-08-09 03:52:38.100285: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47da500 of size 2048 by op Fill action_count 94195480111204 step 0 next 561
    2021-08-09 03:52:38.100303: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47dad00 of size 2048 by op Fill action_count 94195480111206 step 0 next 803
    2021-08-09 03:52:38.100321: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47db500 of size 2048 by op Fill action_count 94195480111207 step 0 next 281
    2021-08-09 03:52:38.100339: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47dbd00 of size 2048 by op Fill action_count 94195480111208 step 0 next 1849
    2021-08-09 03:52:38.100357: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47dc500 of size 2048 by op Fill action_count 94195480111210 step 0 next 1810
    2021-08-09 03:52:38.100375: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47dcd00 of size 2048 by op Fill action_count 94195480111211 step 0 next 535
    2021-08-09 03:52:38.100394: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47dd500 of size 2048 by op Fill action_count 94195480111212 step 0 next 1798
    2021-08-09 03:52:38.100411: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ddd00 of size 2048 by op Fill action_count 94195480111214 step 0 next 2084
    2021-08-09 03:52:38.100425: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47de500 of size 2304 by op Fill action_count 94195480111215 step 0 next 2168
    2021-08-09 03:52:38.100440: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47dee00 of size 256 by op Sub action_count 94195129465161 step 0 next 2177
    2021-08-09 03:52:38.100455: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47def00 of size 256 by op Sub action_count 94195129465162 step 0 next 2086
    2021-08-09 03:52:38.100470: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47df000 of size 3584 by op Fill action_count 94195455013692 step 0 next 415
    2021-08-09 03:52:38.100488: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47dfe00 of size 4096 by op Fill action_count 94195455013694 step 0 next 1246
    2021-08-09 03:52:38.100507: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e0e00 of size 256 by op Sub action_count 94195129465199 step 0 next 989
    2021-08-09 03:52:38.100561: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e0f00 of size 256 by op Sub action_count 94195129465200 step 0 next 2109
    2021-08-09 03:52:38.100592: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e1000 of size 1792 by op Fill action_count 94195455013712 step 0 next 2066
    2021-08-09 03:52:38.100604: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e1700 of size 1792 by op Fill action_count 94195455013714 step 0 next 2933
    2021-08-09 03:52:38.100613: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e1e00 of size 1792 by op Fill action_count 94195455013715 step 0 next 1104
    2021-08-09 03:52:38.100623: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e2500 of size 2816 by op Fill action_count 94195455013716 step 0 next 2454
    2021-08-09 03:52:38.100632: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e3000 of size 256 by op Sub action_count 94195365012667 step 0 next 1811
    2021-08-09 03:52:38.100642: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e3100 of size 256 by op Sub action_count 94195365012668 step 0 next 1998
    2021-08-09 03:52:38.100651: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e3200 of size 1792 by op Fill action_count 94195455013781 step 0 next 602
    2021-08-09 03:52:38.100660: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e3900 of size 1792 by op Fill action_count 94195455013782 step 0 next 214
    2021-08-09 03:52:38.100670: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e4000 of size 1792 by op Fill action_count 94195455013784 step 0 next 1253
    2021-08-09 03:52:38.100679: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e4700 of size 1792 by op Fill action_count 94195455013785 step 0 next 2398
    2021-08-09 03:52:38.100688: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e4e00 of size 1792 by op Fill action_count 94195455013786 step 0 next 1972
    2021-08-09 03:52:38.100697: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e5500 of size 3328 by op Fill action_count 94195455013788 step 0 next 2285
    2021-08-09 03:52:38.100706: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e6200 of size 256 by op Sub action_count 94195339127966 step 0 next 1832
    2021-08-09 03:52:38.100716: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e6300 of size 1024 by op Fill action_count 94195455013408 step 0 next 333
    2021-08-09 03:52:38.100725: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e6700 of size 256 by op Sub action_count 94195339127967 step 0 next 1754
    2021-08-09 03:52:38.100734: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e6800 of size 2048 by op Fill action_count 94195455013454 step 0 next 721
    2021-08-09 03:52:38.100744: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e7000 of size 256 by op Sub action_count 94195339127980 step 0 next 853
    2021-08-09 03:52:38.100753: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e7100 of size 1792 by op Fill action_count 94195455013558 step 0 next 2745
    2021-08-09 03:52:38.100762: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e7800 of size 1792 by op Fill action_count 94195455013559 step 0 next 890
    2021-08-09 03:52:38.100771: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e7f00 of size 2048 by op Fill action_count 94195455013560 step 0 next 2358
    2021-08-09 03:52:38.100780: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e8700 of size 256 by op Sub action_count 94195339127995 step 0 next 845
    2021-08-09 03:52:38.100789: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e8800 of size 2048 by op Fill action_count 94195480110908 step 0 next 2413
    2021-08-09 03:52:38.100798: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e9000 of size 2048 by op Fill action_count 94195480110916 step 0 next 1780
    2021-08-09 03:52:38.100808: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47e9800 of size 2048 by op Fill action_count 94195480110917 step 0 next 61
    2021-08-09 03:52:38.100817: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ea000 of size 2048 by op Fill action_count 94195480110918 step 0 next 865
    2021-08-09 03:52:38.100826: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ea800 of size 2048 by op Fill action_count 94195480110919 step 0 next 1054
    2021-08-09 03:52:38.100835: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47eb000 of size 2048 by op Fill action_count 94195480110920 step 0 next 510
    2021-08-09 03:52:38.100845: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47eb800 of size 4096 by op Fill action_count 94195480110928 step 0 next 1992
    2021-08-09 03:52:38.100854: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ec800 of size 4096 by op Fill action_count 94195480110929 step 0 next 1912
    2021-08-09 03:52:38.100863: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ed800 of size 4096 by op Fill action_count 94195480110930 step 0 next 1156
    2021-08-09 03:52:38.100892: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ee800 of size 4096 by op Fill action_count 94195480110931 step 0 next 167
    2021-08-09 03:52:38.100902: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47ef800 of size 4096 by op Fill action_count 94195480110932 step 0 next 1745
    2021-08-09 03:52:38.100911: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f0800 of size 4096 by op Fill action_count 94195480110940 step 0 next 805
    2021-08-09 03:52:38.100920: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f1800 of size 4096 by op Fill action_count 94195480110941 step 0 next 105
    2021-08-09 03:52:38.100929: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f2800 of size 4096 by op Fill action_count 94195480110942 step 0 next 1729
    2021-08-09 03:52:38.100938: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f3800 of size 4096 by op Fill action_count 94195480110943 step 0 next 1659
    2021-08-09 03:52:38.100947: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f4800 of size 4096 by op Fill action_count 94195480110944 step 0 next 1107
    2021-08-09 03:52:38.100956: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f5800 of size 2048 by op Fill action_count 94195480110952 step 0 next 1086
    2021-08-09 03:52:38.100965: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f6000 of size 2048 by op Fill action_count 94195480110953 step 0 next 353
    2021-08-09 03:52:38.100974: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f6800 of size 2048 by op Fill action_count 94195480110954 step 0 next 556
    2021-08-09 03:52:38.100983: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f7000 of size 2048 by op Fill action_count 94195480110955 step 0 next 1860
    2021-08-09 03:52:38.100992: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f7800 of size 3840 by op Fill action_count 94195480110956 step 0 next 1247
    2021-08-09 03:52:38.101001: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f8700 of size 256 by op Sub action_count 94195129465225 step 0 next 2092
    2021-08-09 03:52:38.101010: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f8800 of size 256 by op Sub action_count 94195129465226 step 0 next 1900
    2021-08-09 03:52:38.101021: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f8900 of size 1792 by op Fill action_count 94195455013752 step 0 next 2603
    2021-08-09 03:52:38.101031: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f9000 of size 1792 by op Fill action_count 94195455013753 step 0 next 1335
    2021-08-09 03:52:38.101041: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f9700 of size 1792 by op Fill action_count 94195455013754 step 0 next 1537
    2021-08-09 03:52:38.101051: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47f9e00 of size 1792 by op Fill action_count 94195455013756 step 0 next 182
    2021-08-09 03:52:38.101061: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47fa500 of size 2304 by op Fill action_count 94195455013757 step 0 next 946
    2021-08-09 03:52:38.101071: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47fae00 of size 256 by op Sub action_count 94195152301760 step 0 next 442
    2021-08-09 03:52:38.101081: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47faf00 of size 1280 by op Fill action_count 94195455013412 step 0 next 274
    2021-08-09 03:52:38.101091: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47fb400 of size 256 by op Sub action_count 94195152301802 step 0 next 966
    2021-08-09 03:52:38.101101: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47fb500 of size 256 by op Sub action_count 94195152301803 step 0 next 110
    2021-08-09 03:52:38.101317: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47fb600 of size 256 by op Sub action_count 94195152301828 step 0 next 1865
    2021-08-09 03:52:38.101375: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae47fb700 of size 7012352 by op Add action_count 94195455013529 step 0 next 781
    2021-08-09 03:52:38.101423: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae4eab700 of size 24661504 by op Fill action_count 94195455013697 step 0 next 614
    2021-08-09 03:52:38.101475: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6630500 of size 256 by op Sub action_count 94195273335853 step 0 next 630
    2021-08-09 03:52:38.101507: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6630600 of size 256 by op Sub action_count 94195273335854 step 0 next 1017
    2021-08-09 03:52:38.101661: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6630700 of size 3584 by op Fill action_count 94195455013695 step 0 next 2225
    2021-08-09 03:52:38.101692: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6631500 of size 4096 by op Fill action_count 94195455013696 step 0 next 559
    2021-08-09 03:52:38.101715: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6632500 of size 256 by op Sub action_count 94195273335867 step 0 next 918
    2021-08-09 03:52:38.101739: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6632600 of size 256 by op Sub action_count 94195273335868 step 0 next 780
    2021-08-09 03:52:38.101765: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6632700 of size 2048 by op Fill action_count 94195455013698 step 0 next 2417
    2021-08-09 03:52:38.101793: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6632f00 of size 2048 by op Fill action_count 94195455013699 step 0 next 1044
    2021-08-09 03:52:38.101819: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6633700 of size 3584 by op Fill action_count 94195455013700 step 0 next 1815
    2021-08-09 03:52:38.101846: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6634500 of size 256 by op Sub action_count 94195273335881 step 0 next 827
    2021-08-09 03:52:38.101871: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6634600 of size 256 by op Sub action_count 94195273335882 step 0 next 1995
    2021-08-09 03:52:38.101895: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6634700 of size 1024 by op Fill action_count 94195467563084 step 0 next 1248
    2021-08-09 03:52:38.101920: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6634b00 of size 1024 by op Fill action_count 94195467563085 step 0 next 1328
    2021-08-09 03:52:38.101946: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6634f00 of size 1024 by op Fill action_count 94195467563086 step 0 next 873
    2021-08-09 03:52:38.101984: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6635300 of size 1024 by op Fill action_count 94195467563087 step 0 next 384
    2021-08-09 03:52:38.102011: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6635700 of size 1024 by op Fill action_count 94195467563095 step 0 next 1325
    2021-08-09 03:52:38.102037: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6635b00 of size 1024 by op Fill action_count 94195467563096 step 0 next 537
    2021-08-09 03:52:38.102062: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6635f00 of size 1024 by op Fill action_count 94195467563097 step 0 next 2634
    2021-08-09 03:52:38.102087: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6636300 of size 1024 by op Fill action_count 94195467563098 step 0 next 1950
    2021-08-09 03:52:38.102114: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6636700 of size 1024 by op Fill action_count 94195467563099 step 0 next 2660
    2021-08-09 03:52:38.102143: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6636b00 of size 256 by op AssignVariableOp action_count 94195480053238 step 0 next 1239
    2021-08-09 03:52:38.102171: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6636c00 of size 256 by op AssignVariableOp action_count 94195480053240 step 0 next 478
    2021-08-09 03:52:38.102197: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6636d00 of size 256 by op Sub action_count 94195480110994 step 0 next 821
    2021-08-09 03:52:38.102240: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6636e00 of size 256 by op AssignAddVariableOp_12 action_count 94195480109429 step 17368542405085625385 next 778
    2021-08-09 03:52:38.102270: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6636f00 of size 256 by op Sub action_count 94195480110995 step 0 next 1693
    2021-08-09 03:52:38.102299: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6637000 of size 256 by op AssignVariableOp action_count 94195480053234 step 0 next 1852
    2021-08-09 03:52:38.102327: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6637100 of size 256 by op AssignVariableOp action_count 94195480053236 step 0 next 1129
    2021-08-09 03:52:38.102352: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6637200 of size 256 by op Sub action_count 94195480111032 step 0 next 1961
    2021-08-09 03:52:38.102378: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6637300 of size 1024 by op Add action_count 94195467563104 step 0 next 2418
    2021-08-09 03:52:38.102405: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6637700 of size 256 by op Sub action_count 94195480111033 step 0 next 2315
    2021-08-09 03:52:38.102429: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6637800 of size 256 by op AssignVariableOp action_count 94195480053226 step 0 next 1180
    2021-08-09 03:52:38.102466: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6637900 of size 256 by op Sub action_count 94195480111070 step 0 next 710
    2021-08-09 03:52:38.102495: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6637a00 of size 256 by op AssignAddVariableOp_10 action_count 94195480109361 step 17368542405085625385 next 2555
    2021-08-09 03:52:38.102540: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6637b00 of size 1024 by op Fill action_count 94195480110850 step 0 next 1888
    2021-08-09 03:52:38.102570: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6637f00 of size 1280 by op Fill action_count 94195480110851 step 0 next 2317
    2021-08-09 03:52:38.102595: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6638400 of size 256 by op AssignAddVariableOp_52 action_count 94195480053815 step 12509324397843644032 next 153
    2021-08-09 03:52:38.102624: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6638500 of size 512 by op Fill action_count 94195480110836 step 0 next 835
    2021-08-09 03:52:38.102652: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6638700 of size 768 by op Fill action_count 94195480110837 step 0 next 958
    2021-08-09 03:52:38.102683: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6638a00 of size 256 by op AssignAddVariableOp_20 action_count 94195480109433 step 17368542405085625385 next 1113
    2021-08-09 03:52:38.102710: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6638b00 of size 256 by op AssignAddVariableOp_18 action_count 94195480109373 step 17368542405085625385 next 292
    2021-08-09 03:52:38.102742: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6638c00 of size 256 by op AssignAddVariableOp_28 action_count 94195480109437 step 17368542405085625385 next 610
    2021-08-09 03:52:38.102778: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6638d00 of size 256 by op AssignVariableOp action_count 94195480053198 step 0 next 226
    2021-08-09 03:52:38.102804: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6638e00 of size 256 by op AssignVariableOp action_count 94195480053204 step 0 next 1951
    2021-08-09 03:52:38.102844: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6638f00 of size 256 by op AssignAddVariableOp_11 action_count 94195480109363 step 17368542405085625385 next 2366
    2021-08-09 03:52:38.102873: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6639000 of size 256 by op Sub action_count 94195480111071 step 0 next 2282
    2021-08-09 03:52:38.102898: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6639100 of size 256 by op AssignVariableOp action_count 94195479483519 step 0 next 299
    2021-08-09 03:52:38.102926: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6639200 of size 256 by op AssignVariableOp action_count 94195480053200 step 0 next 1839
    2021-08-09 03:52:38.102955: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6639300 of size 256 by op Fill action_count 94195480111102 step 0 next 1398
    2021-08-09 03:52:38.102990: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6639400 of size 256 by op ConstantFolding/truediv_recip action_count 94195468133239 step 0 next 2053
    2021-08-09 03:52:38.103020: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6639500 of size 256 by op add_16/y action_count 94195468133240 step 0 next 2983
    2021-08-09 03:52:38.103048: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6639600 of size 256 by op Sum_1 action_count 94195468133241 step 0 next 1948
    2021-08-09 03:52:38.103075: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6639700 of size 256 by op truediv/y action_count 94195468133243 step 0 next 1020
    2021-08-09 03:52:38.103099: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6639800 of size 256 by op binary_ce_dice/add_1/y action_count 94195468133244 step 0 next 1875
    2021-08-09 03:52:38.103134: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6639900 of size 256 by op AssignVariableOp action_count 94195480053248 step 0 next 436
    2021-08-09 03:52:38.103163: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6639a00 of size 256 by op AssignVariableOp action_count 94195480053216 step 0 next 1527
    2021-08-09 03:52:38.103186: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6639b00 of size 256 by op Fill action_count 94195480111103 step 0 next 1241
    2021-08-09 03:52:38.103210: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6639c00 of size 256 by op Fill action_count 94195467563161 step 0 next 1855
    2021-08-09 03:52:38.103236: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6639d00 of size 1536 by op Fill action_count 94195467563162 step 0 next 487
    2021-08-09 03:52:38.103263: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663a300 of size 512 by op Fill action_count 94195467563163 step 0 next 2496
    2021-08-09 03:52:38.103296: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663a500 of size 512 by op Fill action_count 94195467563164 step 0 next 969
    2021-08-09 03:52:38.103324: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663a700 of size 512 by op Fill action_count 94195467563165 step 0 next 1968
    2021-08-09 03:52:38.103346: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663a900 of size 512 by op Fill action_count 94195467563167 step 0 next 1922
    2021-08-09 03:52:38.103371: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663ab00 of size 512 by op Fill action_count 94195467563168 step 0 next 698
    2021-08-09 03:52:38.103397: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663ad00 of size 512 by op Fill action_count 94195467563169 step 0 next 1397
    2021-08-09 03:52:38.103423: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663af00 of size 1024 by op Fill action_count 94195467563171 step 0 next 1202
    2021-08-09 03:52:38.103450: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663b300 of size 1024 by op Fill action_count 94195467563172 step 0 next 2420
    2021-08-09 03:52:38.103484: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663b700 of size 1024 by op Fill action_count 94195467563173 step 0 next 194
    2021-08-09 03:52:38.103512: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663bb00 of size 1024 by op Fill action_count 94195467563175 step 0 next 2276
    2021-08-09 03:52:38.103546: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663bf00 of size 1024 by op Fill action_count 94195467563176 step 0 next 2593
    2021-08-09 03:52:38.103571: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663c300 of size 1024 by op Fill action_count 94195467563177 step 0 next 1395
    2021-08-09 03:52:38.103593: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663c700 of size 1792 by op Fill action_count 94195467563179 step 0 next 217
    2021-08-09 03:52:38.103618: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663ce00 of size 1792 by op Fill action_count 94195467563180 step 0 next 804
    2021-08-09 03:52:38.103664: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663d500 of size 1792 by op Fill action_count 94195467563181 step 0 next 1966
    2021-08-09 03:52:38.103693: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663dc00 of size 1792 by op Fill action_count 94195467563183 step 0 next 1491
    2021-08-09 03:52:38.103719: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663e300 of size 256 by op Sub action_count 94195273335919 step 0 next 1963
    2021-08-09 03:52:38.103746: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663e400 of size 256 by op Sub action_count 94195273335920 step 0 next 424
    2021-08-09 03:52:38.103774: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663e500 of size 3584 by op Fill action_count 94195455013765 step 0 next 963
    2021-08-09 03:52:38.103800: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae663f300 of size 6656 by op Fill action_count 94195455013766 step 0 next 867
    2021-08-09 03:52:38.103835: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6640d00 of size 256 by op Sub action_count 94195273335957 step 0 next 920
    2021-08-09 03:52:38.103864: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6640e00 of size 256 by op Sub action_count 94195273335958 step 0 next 1318
    2021-08-09 03:52:38.103891: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6640f00 of size 1792 by op Fill action_count 94195455013561 step 0 next 85
    2021-08-09 03:52:38.103917: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6641600 of size 1792 by op Fill action_count 94195455013562 step 0 next 1266
    2021-08-09 03:52:38.103943: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6641d00 of size 1024 by op Fill action_count 94195455013570 step 0 next 1684
    2021-08-09 03:52:38.103968: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6642100 of size 1024 by op Fill action_count 94195455013571 step 0 next 211
    2021-08-09 03:52:38.104005: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6642500 of size 256 by op Sub action_count 94195273335983 step 0 next 841
    2021-08-09 03:52:38.104031: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6642600 of size 256 by op Sub action_count 94195273335984 step 0 next 1782
    2021-08-09 03:52:38.104062: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6642700 of size 8192000 by op Tile_90/_0__cf__3862 action_count 94195410013342 step 0 next 1480
    2021-08-09 03:52:38.104090: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6e12700 of size 1099264 by op Fill action_count 94195455013681 step 0 next 2749
    2021-08-09 03:52:38.104118: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6f1ed00 of size 549632 by op Fill action_count 94195455013747 step 0 next 1434
    2021-08-09 03:52:38.104146: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae6fa5000 of size 1055488 by op Fill action_count 94195455013795 step 0 next 2865
    2021-08-09 03:52:38.104181: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae70a6b00 of size 256 by op AssignAddVariableOp_52 action_count 94195410014265 step 11906430663947522641 next 2684
    2021-08-09 03:52:38.104211: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae70a6c00 of size 8192000 by op Tile_90/_0__cf__3864 action_count 94195412081363 step 0 next 2864
    2021-08-09 03:52:38.104238: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae7876c00 of size 8792832 by op Fill action_count 94195467563190 step 0 next 2262
    2021-08-09 03:52:38.104266: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae80d9700 of size 8792832 by op Add action_count 94195467562984 step 0 next 2484
    2021-08-09 03:52:38.104292: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae893c200 of size 5775360 by op Add action_count 94195467563008 step 0 next 888
    2021-08-09 03:52:38.104329: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae8ebe200 of size 3297536 by op Add action_count 94195467563056 step 0 next 308
    2021-08-09 03:52:38.104359: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae91e3300 of size 3714816 by op Add action_count 94195467563044 step 0 next 1489
    2021-08-09 03:52:38.104385: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae956e200 of size 7012352 by op Add action_count 94195467563032 step 0 next 2216
    2021-08-09 03:52:38.104411: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae9c1e200 of size 2930944 by op Fill action_count 94195467563182 step 0 next 1615
    2021-08-09 03:52:38.104449: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ae9ee9b00 of size 5318400 by op Add action_count 94195467563068 step 0 next 2220
    2021-08-09 03:52:38.104478: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aea3fc200 of size 14024704 by op Add action_count 94195467562996 step 0 next 1230
    2021-08-09 03:52:38.104504: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aeb15c200 of size 14024704 by op Fill action_count 94195467563194 step 0 next 1924
    2021-08-09 03:52:38.104538: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aebebc200 of size 5775360 by op Fill action_count 94195467563198 step 0 next 875
    2021-08-09 03:52:38.104577: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aec43e200 of size 3145728 by op Fill action_count 94195467563202 step 0 next 1655
    2021-08-09 03:52:38.104607: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aec73e200 of size 7012352 by op Fill action_count 94195467563206 step 0 next 940
    2021-08-09 03:52:38.104635: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aecdee200 of size 3297536 by op Fill action_count 94195467563210 step 0 next 325
    2021-08-09 03:52:38.104661: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aed113300 of size 2198272 by op Fill action_count 94195467563214 step 0 next 1562
    2021-08-09 03:52:38.104695: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aed32be00 of size 2930944 by op Fill action_count 94195467563218 step 0 next 2478
    2021-08-09 03:52:38.104732: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aed5f7700 of size 824576 by op Fill action_count 94195467563222 step 0 next 1762
    2021-08-09 03:52:38.104761: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aed6c0c00 of size 274944 by op Fill action_count 94195467563240 step 0 next 990
    2021-08-09 03:52:38.104787: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aed703e00 of size 549632 by op Fill action_count 94195467563244 step 0 next 2485
    2021-08-09 03:52:38.104813: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aed78a100 of size 1099264 by op Fill action_count 94195467563248 step 0 next 646
    2021-08-09 03:52:38.104838: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aed896700 of size 2198272 by op Fill action_count 94195467563252 step 0 next 359
    2021-08-09 03:52:38.104876: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aedaaf200 of size 4396544 by op Fill action_count 94195467563256 step 0 next 480
    2021-08-09 03:52:38.104902: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aedee0800 of size 8792832 by op Fill action_count 94195467563260 step 0 next 1417
    2021-08-09 03:52:38.104929: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aee743300 of size 14024704 by op Fill action_count 94195467563264 step 0 next 2389
    2021-08-09 03:52:38.104957: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aef4a3300 of size 5775360 by op Fill action_count 94195467563268 step 0 next 1738
    2021-08-09 03:52:38.104985: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aefa25300 of size 3145728 by op Fill action_count 94195467563272 step 0 next 207
    2021-08-09 03:52:38.105021: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3aefd25300 of size 7012352 by op Fill action_count 94195467563276 step 0 next 1300
    2021-08-09 03:52:38.105048: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af03d5300 of size 3297536 by op Fill action_count 94195467563280 step 0 next 2101
    2021-08-09 03:52:38.105074: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af06fa400 of size 2198272 by op Fill action_count 94195467563284 step 0 next 2381
    2021-08-09 03:52:38.105102: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af0912f00 of size 2930944 by op Fill action_count 94195467563288 step 0 next 790
    2021-08-09 03:52:38.105128: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af0bde800 of size 824576 by op Fill action_count 94195467563292 step 0 next 759
    2021-08-09 03:52:38.105165: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af0ca7d00 of size 549632 by op Fill action_count 94195467563296 step 0 next 2598
    2021-08-09 03:52:38.105196: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af0d2e000 of size 425984 by op gradient_tape/binary_ce_dice/logistic_loss/zeros_like action_count 94195467563312 step 0 next 1368
    2021-08-09 03:52:38.105224: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af0d96000 of size 425984 by op Tile_36 action_count 94195467563317 step 0 next 1772
    2021-08-09 03:52:38.105253: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af0dfe000 of size 425984 by op Tile_54 action_count 94195467563318 step 0 next 2232
    2021-08-09 03:52:38.105289: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af0e66000 of size 425984 by op Tile_72 action_count 94195467563319 step 0 next 1208
    2021-08-09 03:52:38.105321: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af0ece000 of size 425984 by op Tile_93 action_count 94195467563321 step 0 next 704
    2021-08-09 03:52:38.105350: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af0f36000 of size 425984 by op Tile_18 action_count 94195467563322 step 0 next 534
    2021-08-09 03:52:38.105377: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af0f9e000 of size 425984 by op Tile action_count 94195467563323 step 0 next 216
    2021-08-09 03:52:38.105416: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af1006000 of size 425984 by op gradient_tape/binary_ce_dice/logistic_loss/sub/Neg/_0__cf__4148 action_count 94195467563478 step 0 next 1578
    2021-08-09 03:52:38.105447: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af106e000 of size 425984 by op gradient_tape/binary_ce_dice/weighted_loss/Tile_1/_1__cf__4149 action_count 94195467563479 step 0 next 1495
    2021-08-09 03:52:38.105474: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af10d6000 of size 425984 by op binary_ce_dice/logistic_loss/zeros_like action_count 94195468133242 step 0 next 1119
    2021-08-09 03:52:38.105501: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af113e000 of size 425984 by op Tile action_count 94195468133246 step 0 next 372
    2021-08-09 03:52:38.105544: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af11a6000 of size 425984 by op Tile_93 action_count 94195468133248 step 0 next 1447
    2021-08-09 03:52:38.105573: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af120e000 of size 425984 by op Tile_18 action_count 94195468133249 step 0 next 142
    2021-08-09 03:52:38.105601: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af1276000 of size 425984 by op Tile_36 action_count 94195468133250 step 0 next 48
    2021-08-09 03:52:38.105627: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af12de000 of size 425984 by op Tile_54 action_count 94195468133251 step 0 next 554
    2021-08-09 03:52:38.105661: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af1346000 of size 425984 by op Tile_72 action_count 94195468133252 step 0 next 2450
    2021-08-09 03:52:38.105688: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af13ae000 of size 167168 by op Fill action_count 94195480111161 step 0 next 1695
    2021-08-09 03:52:38.105715: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af13d6d00 of size 167168 by op Fill action_count 94195480111251 step 0 next 801
    2021-08-09 03:52:38.105741: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af13ffa00 of size 167168 by op Add action_count 94195480110833 step 0 next 1896
    2021-08-09 03:52:38.105779: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af1428700 of size 334336 by op Fill action_count 94195480111165 step 0 next 2100
    2021-08-09 03:52:38.105806: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af147a100 of size 334336 by op Add action_count 94195480110847 step 0 next 243
    2021-08-09 03:52:38.105832: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af14cbb00 of size 668416 by op Fill action_count 94195480111169 step 0 next 1256
    2021-08-09 03:52:38.105854: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af156ee00 of size 668416 by op Add action_count 94195480111087 step 0 next 885
    2021-08-09 03:52:38.105891: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af1612100 of size 668416 by op Add action_count 94195480110861 step 0 next 69
    2021-08-09 03:52:38.105917: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af16b5400 of size 1336832 by op Add action_count 94195480111075 step 0 next 522
    2021-08-09 03:52:38.105943: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af17fba00 of size 1336832 by op Add action_count 94195480110875 step 0 next 2605
    2021-08-09 03:52:38.105970: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af1942000 of size 2673408 by op Add action_count 94195480111049 step 0 next 1528
    2021-08-09 03:52:38.106007: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af1bceb00 of size 2673408 by op Fill action_count 94195480111173 step 0 next 420
    2021-08-09 03:52:38.106037: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af1e5b600 of size 2673408 by op Add action_count 94195480110887 step 0 next 1731
    2021-08-09 03:52:38.106064: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af20e8100 of size 2899968 by op Fill action_count 94195480111177 step 0 next 2014
    2021-08-09 03:52:38.106090: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af23ac100 of size 2899968 by op Add action_count 94195480110901 step 0 next 997
    2021-08-09 03:52:38.106127: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af2670100 of size 3145728 by op Add action_count 94195480111023 step 0 next 2104
    2021-08-09 03:52:38.106156: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af2970100 of size 3145728 by op Add action_count 94195480110913 step 0 next 90
    2021-08-09 03:52:38.106184: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af2c70100 of size 2097152 by op Add action_count 94195480110985 step 0 next 346
    2021-08-09 03:52:38.106211: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af2e70100 of size 891136 by op Add action_count 94195480111061 step 0 next 51
    2021-08-09 03:52:38.106248: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af2f49a00 of size 1206016 by op Fill action_count 94195480111233 step 0 next 651
    2021-08-09 03:52:38.106275: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af3070100 of size 4194304 by op Add action_count 94195480110949 step 0 next 1880
    2021-08-09 03:52:38.106301: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af3470100 of size 4194304 by op Add action_count 94195480110973 step 0 next 125
    2021-08-09 03:52:38.106338: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af3870100 of size 6291456 by op Add action_count 94195480110925 step 0 next 787
    2021-08-09 03:52:38.106367: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af3e70100 of size 3145728 by op Add action_count 94195480111011 step 0 next 1250
    2021-08-09 03:52:38.106396: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af4170100 of size 4010240 by op Fill action_count 94195480111181 step 0 next 282
    2021-08-09 03:52:38.106433: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af4543200 of size 5426944 by op Add action_count 94195480111037 step 0 next 1109
    2021-08-09 03:52:38.106463: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af4a70100 of size 14162944 by op Add action_count 94195480110937 step 0 next 2163
    2021-08-09 03:52:38.106489: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af57f1d00 of size 256 by op Sub action_count 94195129464775 step 0 next 135
    2021-08-09 03:52:38.106531: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af57f1e00 of size 1099264 by op Add action_count 94195455013433 step 0 next 2920
    2021-08-09 03:52:38.106562: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af58fe400 of size 2723840 by op Fill action_count 94195455013685 step 0 next 1741
    2021-08-09 03:52:38.106588: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b97400 of size 256 by op Sub action_count 94195347535728 step 0 next 312
    2021-08-09 03:52:38.106624: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b97500 of size 512 by op Reshape_92 action_count 94195455584276 step 0 next 1165
    2021-08-09 03:52:38.106653: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b97700 of size 256 by op Cast_181/x action_count 94195455584282 step 0 next 1894
    2021-08-09 03:52:38.106685: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b97800 of size 256 by op unet_depth3/encode0/batch_normalization_546/batchnorm/add/y action_count 94195455584283 step 0 next 2269
    2021-08-09 03:52:38.106722: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b97900 of size 256 by op Cast_180/x action_count 94195455584284 step 0 next 2460
    2021-08-09 03:52:38.106751: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b97a00 of size 256 by op assert_greater_equal/Assert/AssertGuard/pivot_f/_3 action_count 94195455584285 step 0 next 669
    2021-08-09 03:52:38.106779: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b97b00 of size 256 by op assert_greater_equal/Assert/AssertGuard/pivot_t/_4 action_count 94195455584286 step 0 next 2847
    2021-08-09 03:52:38.106814: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b97c00 of size 256 by op Const_35 action_count 94195455584287 step 0 next 2542
    2021-08-09 03:52:38.106843: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b97d00 of size 256 by op assert_less_equal/Assert/AssertGuard/pivot_f/_13 action_count 94195455584288 step 0 next 2314
    2021-08-09 03:52:38.106870: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b97e00 of size 256 by op assert_less_equal/Assert/AssertGuard/pivot_t/_14 action_count 94195455584289 step 0 next 2364
    2021-08-09 03:52:38.106897: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b97f00 of size 256 by op assert_greater_equal_1/Assert/AssertGuard/pivot_f/_31 action_count 94195455584290 step 0 next 2687
    2021-08-09 03:52:38.106933: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b98000 of size 256 by op assert_greater_equal_1/Assert/AssertGuard/pivot_t/_32 action_count 94195455584291 step 0 next 2565
    2021-08-09 03:52:38.106963: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b98100 of size 256 by op assert_less_equal_1/Assert/AssertGuard/pivot_f/_41 action_count 94195455584292 step 0 next 2320
    2021-08-09 03:52:38.106990: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b98200 of size 256 by op assert_less_equal_1/Assert/AssertGuard/pivot_t/_42 action_count 94195455584293 step 0 next 1703
    2021-08-09 03:52:38.107017: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b98300 of size 256 by op assert_greater_equal_2/Assert/AssertGuard/pivot_f/_59 action_count 94195455584294 step 0 next 2452
    2021-08-09 03:52:38.107053: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b98400 of size 256 by op assert_greater_equal_2/Assert/AssertGuard/pivot_t/_60 action_count 94195455584295 step 0 next 3058
    2021-08-09 03:52:38.107081: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b98500 of size 256 by op AssignVariableOp action_count 94195467505291 step 0 next 2676
    2021-08-09 03:52:38.107107: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b98600 of size 256 by op AssignVariableOp action_count 94195480053202 step 0 next 2558
    2021-08-09 03:52:38.107129: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b98700 of size 256 by op AssignVariableOp action_count 94195467505325 step 0 next 779
    2021-08-09 03:52:38.107162: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b98800 of size 256 by op AssignVariableOp action_count 94195467505327 step 0 next 2591
    2021-08-09 03:52:38.107191: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b98900 of size 512 by op AssignVariableOp action_count 94195467505337 step 0 next 797
    2021-08-09 03:52:38.107217: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b98b00 of size 768 by op Fill action_count 94195467562906 step 0 next 716
    2021-08-09 03:52:38.107243: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b98e00 of size 256 by op Sub action_count 94195347535757 step 0 next 485
    2021-08-09 03:52:38.107265: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b98f00 of size 256 by op Sub action_count 94195347535770 step 0 next 233
    2021-08-09 03:52:38.107301: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b99000 of size 256 by op Sub action_count 94195347535771 step 0 next 1231
    2021-08-09 03:52:38.107328: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af5b99100 of size 6291456 by op Add action_count 94195480110961 step 0 next 1088
    2021-08-09 03:52:38.107355: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af6199100 of size 6045696 by op Add action_count 94195480110999 step 0 next 445
    2021-08-09 03:52:38.107382: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af675d100 of size 3145728 by op Fill action_count 94195480111185 step 0 next 460
    2021-08-09 03:52:38.107408: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af6a5d100 of size 6291456 by op Fill action_count 94195480111189 step 0 next 78
    2021-08-09 03:52:38.107445: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af705d100 of size 12582912 by op Fill action_count 94195480111193 step 0 next 1485
    2021-08-09 03:52:38.107473: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af7c5d100 of size 4194304 by op Fill action_count 94195480111197 step 0 next 694
    2021-08-09 03:52:38.107499: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af805d100 of size 6291456 by op Fill action_count 94195480111201 step 0 next 1071
    2021-08-09 03:52:38.107528: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af865d100 of size 3145728 by op Fill action_count 94195480111205 step 0 next 1285
    2021-08-09 03:52:38.107563: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af895d100 of size 2097152 by op Fill action_count 94195480111209 step 0 next 627
    2021-08-09 03:52:38.107592: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af8b5d100 of size 6045696 by op Fill action_count 94195480111213 step 0 next 1817
    2021-08-09 03:52:38.107620: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af9121100 of size 3145728 by op Fill action_count 94195480111217 step 0 next 531
    2021-08-09 03:52:38.107648: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af9421100 of size 1933312 by op Fill action_count 94195480111221 step 0 next 1690
    2021-08-09 03:52:38.107685: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af95f9100 of size 4010240 by op Fill action_count 94195480111225 step 0 next 34
    2021-08-09 03:52:38.107714: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af99cc200 of size 2673408 by op Fill action_count 94195480111229 step 0 next 2339
    2021-08-09 03:52:38.107744: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af9c58d00 of size 1002752 by op Fill action_count 94195480111237 step 0 next 202
    2021-08-09 03:52:38.107771: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af9d4da00 of size 668416 by op Fill action_count 94195480111241 step 0 next 1126
    2021-08-09 03:52:38.107800: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af9df0d00 of size 334336 by op Fill action_count 94195480111255 step 0 next 753
    2021-08-09 03:52:38.107827: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af9e42700 of size 668416 by op Fill action_count 94195480111259 step 0 next 99
    2021-08-09 03:52:38.107854: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3af9ee5a00 of size 1336832 by op Fill action_count 94195480111263 step 0 next 70
    2021-08-09 03:52:38.107880: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afa02c000 of size 2673408 by op Fill action_count 94195480111267 step 0 next 483
    2021-08-09 03:52:38.107904: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afa2b8b00 of size 2899968 by op Fill action_count 94195480111271 step 0 next 1603
    2021-08-09 03:52:38.107926: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afa57cb00 of size 3145728 by op Fill action_count 94195480111275 step 0 next 1661
    2021-08-09 03:52:38.107950: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afa87cb00 of size 6291456 by op Fill action_count 94195480111279 step 0 next 1123
    2021-08-09 03:52:38.107988: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afae7cb00 of size 12582912 by op Fill action_count 94195480111283 step 0 next 92
    2021-08-09 03:52:38.108018: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afba7cb00 of size 4194304 by op Fill action_count 94195480111287 step 0 next 1326
    2021-08-09 03:52:38.108046: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afbe7cb00 of size 6291456 by op Fill action_count 94195480111291 step 0 next 695
    2021-08-09 03:52:38.108073: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afc47cb00 of size 3145728 by op Fill action_count 94195480111295 step 0 next 626
    2021-08-09 03:52:38.108099: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afc77cb00 of size 2097152 by op Fill action_count 94195480111299 step 0 next 1067
    2021-08-09 03:52:38.108126: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afc97cb00 of size 6045696 by op Fill action_count 94195480111303 step 0 next 1103
    2021-08-09 03:52:38.108162: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afcf40b00 of size 2048 by op Fill action_count 94195480111305 step 0 next 472
    2021-08-09 03:52:38.108190: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afcf41300 of size 2048 by op Fill action_count 94195480111306 step 0 next 1367
    2021-08-09 03:52:38.108216: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afcf41b00 of size 3145728 by op Fill action_count 94195480111307 step 0 next 750
    2021-08-09 03:52:38.108238: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afd241b00 of size 2048 by op Fill action_count 94195480111308 step 0 next 286
    2021-08-09 03:52:38.108262: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afd242300 of size 2048 by op Fill action_count 94195480111309 step 0 next 658
    2021-08-09 03:52:38.108284: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afd242b00 of size 2048 by op Fill action_count 94195480111310 step 0 next 1090
    2021-08-09 03:52:38.108316: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afd243300 of size 1933312 by op Fill action_count 94195480111311 step 0 next 370
    2021-08-09 03:52:38.108343: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afd41b300 of size 2048 by op Fill action_count 94195480111312 step 0 next 1555
    2021-08-09 03:52:38.108369: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afd41bb00 of size 2048 by op Fill action_count 94195480111313 step 0 next 947
    2021-08-09 03:52:38.108394: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afd41c300 of size 2048 by op Fill action_count 94195480111314 step 0 next 1504
    2021-08-09 03:52:38.108421: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afd41cb00 of size 4010240 by op Fill action_count 94195480111315 step 0 next 1336
    2021-08-09 03:52:38.108450: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afd7efc00 of size 2048 by op Fill action_count 94195480111316 step 0 next 492
    2021-08-09 03:52:38.108476: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afd7f0400 of size 2048 by op Fill action_count 94195480111317 step 0 next 1171
    2021-08-09 03:52:38.108503: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afd7f0c00 of size 2048 by op Fill action_count 94195480111318 step 0 next 469
    2021-08-09 03:52:38.108541: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afd7f1400 of size 2673408 by op Fill action_count 94195480111319 step 0 next 1529
    2021-08-09 03:52:38.108570: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afda7df00 of size 2048 by op Fill action_count 94195480111320 step 0 next 948
    2021-08-09 03:52:38.108607: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afda7e700 of size 2048 by op Fill action_count 94195480111321 step 0 next 1599
    2021-08-09 03:52:38.108636: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afda7ef00 of size 2048 by op Fill action_count 94195480111322 step 0 next 1009
    2021-08-09 03:52:38.108662: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afda7f700 of size 891136 by op Fill action_count 94195480111323 step 0 next 569
    2021-08-09 03:52:38.108689: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdb59000 of size 1024 by op Fill action_count 94195480111324 step 0 next 1520
    2021-08-09 03:52:38.108717: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdb59400 of size 1024 by op Fill action_count 94195480111325 step 0 next 463
    2021-08-09 03:52:38.108745: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdb59800 of size 1024 by op Fill action_count 94195480111326 step 0 next 1293
    2021-08-09 03:52:38.108780: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdb59c00 of size 1002752 by op Fill action_count 94195480111327 step 0 next 1486
    2021-08-09 03:52:38.108808: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdc4e900 of size 1024 by op Fill action_count 94195480111328 step 0 next 1642
    2021-08-09 03:52:38.108830: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdc4ed00 of size 1024 by op Fill action_count 94195480111329 step 0 next 296
    2021-08-09 03:52:38.108854: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdc4f100 of size 1024 by op Fill action_count 94195480111330 step 0 next 654
    2021-08-09 03:52:38.108882: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdc4f500 of size 668416 by op Fill action_count 94195480111331 step 0 next 187
    2021-08-09 03:52:38.108908: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf2800 of size 1024 by op Fill action_count 94195480111332 step 0 next 1851
    2021-08-09 03:52:38.108934: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf2c00 of size 1024 by op Fill action_count 94195480111333 step 0 next 713
    2021-08-09 03:52:38.108962: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf3000 of size 1024 by op Fill action_count 94195480111334 step 0 next 116
    2021-08-09 03:52:38.108995: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf3400 of size 1024 by op Fill action_count 94195480111335 step 0 next 2017
    2021-08-09 03:52:38.109022: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf3800 of size 256 by op Fill action_count 94195480111336 step 0 next 1222
    2021-08-09 03:52:38.109058: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf3900 of size 256 by op ConstantFolding/truediv_recip action_count 94195480111337 step 0 next 1538
    2021-08-09 03:52:38.109088: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf3a00 of size 256 by op ConstantFolding/gradient_tape/unet_depth4/encode0/batch_normalization_580/moments/truediv_recip action_count 94
    195480111338 step 0 next 563
    2021-08-09 03:52:38.109118: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf3b00 of size 256 by op ConstantFolding/gradient_tape/unet_depth4/encode1/batch_normalization_582/moments/truediv_recip action_count 94
    195480111339 step 0 next 1699
    2021-08-09 03:52:38.109148: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf3c00 of size 256 by op ConstantFolding/gradient_tape/unet_depth4/encode2/batch_normalization_584/moments/truediv_recip action_count 94
    195480111340 step 0 next 960
    2021-08-09 03:52:38.109178: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf3d00 of size 256 by op ConstantFolding/gradient_tape/unet_depth4/encode3/batch_normalization_586/moments/truediv_recip action_count 94
    195480111341 step 0 next 1704
    2021-08-09 03:52:38.109207: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf3e00 of size 256 by op ConstantFolding/gradient_tape/unet_depth4/two_conv_center/batch_normalization_588/moments/truediv_recip action_
    count 94195480111342 step 0 next 1345
    2021-08-09 03:52:38.109234: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf3f00 of size 256 by op Adam/add/y action_count 94195480111343 step 0 next 1887
    2021-08-09 03:52:38.109261: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf4000 of size 256 by op add_16/y action_count 94195480111344 step 0 next 1579
    2021-08-09 03:52:38.109298: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf4100 of size 256 by op gradient_tape/binary_ce_dice/weighted_loss/Reshape_1 action_count 94195480111345 step 0 next 2117
    2021-08-09 03:52:38.109329: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf4200 of size 256 by op gradient_tape/binary_ce_dice/logistic_loss/zeros_like/Const action_count 94195480111346 step 0 next 690
    2021-08-09 03:52:38.109355: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf4300 of size 256 by op Sum_1 action_count 94195480111347 step 0 next 1701
    2021-08-09 03:52:38.109391: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdcf4400 of size 1441792 by op gradient_tape/binary_ce_dice/logistic_loss/zeros_like action_count 94195480111348 step 0 next 2114
    2021-08-09 03:52:38.109420: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afde54400 of size 256 by op Adam/gradients/ones action_count 94195480111349 step 0 next 1058
    2021-08-09 03:52:38.109447: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afde54500 of size 256 by op truediv/y action_count 94195480111350 step 0 next 1822
    2021-08-09 03:52:38.109473: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afde54600 of size 256 by op binary_ce_dice/add_1/y action_count 94195480111351 step 0 next 17
    2021-08-09 03:52:38.109510: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afde54700 of size 1441792 by op Tile_93 action_count 94195480111352 step 0 next 223
    2021-08-09 03:52:38.109544: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdfb4700 of size 256 by op binary_ce_dice/weighted_loss/num_elements/Cast action_count 94195480111353 step 0 next 2135
    2021-08-09 03:52:38.109571: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afdfb4800 of size 1441792 by op Tile_18 action_count 94195480111354 step 0 next 692
    2021-08-09 03:52:38.109597: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe114800 of size 1441792 by op Tile_36 action_count 94195480111355 step 0 next 1499
    2021-08-09 03:52:38.109619: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe274800 of size 1441792 by op Tile_54 action_count 94195480111356 step 0 next 1747
    2021-08-09 03:52:38.109644: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe3d4800 of size 1441792 by op Tile_72 action_count 94195480111357 step 0 next 1220
    2021-08-09 03:52:38.109672: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe534800 of size 1441792 by op Tile action_count 94195480111358 step 0 next 1840
    2021-08-09 03:52:38.109709: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe694800 of size 512 by op Reshape_92 action_count 94195480111359 step 0 next 941
    2021-08-09 03:52:38.109740: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe694a00 of size 256 by op unet_depth4/conv_transpose_decoder0/batch_normalization_599/AssignMovingAvg/decay action_count 94195480111360 s
    tep 0 next 283
    2021-08-09 03:52:38.109767: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe694b00 of size 256 by op unet_depth4/two_conv_decoder0/batch_normalization_601/batchnorm/add/y action_count 94195480111361 step 0 next 3
    86
    2021-08-09 03:52:38.109794: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe694c00 of size 256 by op gradient_tape/binary_ce_dice/logistic_loss/add/x action_count 94195480111362 step 0 next 1243
    2021-08-09 03:52:38.109820: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe694d00 of size 256 by op assert_greater_equal/Assert/AssertGuard/pivot_f/_3 action_count 94195480111363 step 0 next 2145
    2021-08-09 03:52:38.109847: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe694e00 of size 256 by op assert_greater_equal/Assert/AssertGuard/pivot_t/_4 action_count 94195480111364 step 0 next 1275
    2021-08-09 03:52:38.109884: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe694f00 of size 256 by op assert_less_equal/Assert/AssertGuard/pivot_f/_13 action_count 94195480111365 step 0 next 1304
    2021-08-09 03:52:38.109912: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695000 of size 256 by op assert_less_equal/Assert/AssertGuard/pivot_t/_14 action_count 94195480111366 step 0 next 1317
    2021-08-09 03:52:38.109938: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695100 of size 256 by op Const_35 action_count 94195480111367 step 0 next 720
    2021-08-09 03:52:38.109965: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695200 of size 256 by op ArithmeticOptimizer/AddOpsRewrite_add_16/tmp_var_zeros action_count 94195480111368 step 0 next 1796
    2021-08-09 03:52:38.109991: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695300 of size 256 by op assert_greater_equal_1/Assert/AssertGuard/pivot_f/_31 action_count 94195480111369 step 0 next 1288
    2021-08-09 03:52:38.110018: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695400 of size 256 by op assert_greater_equal_1/Assert/AssertGuard/pivot_t/_32 action_count 94195480111370 step 0 next 1833
    2021-08-09 03:52:38.110052: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695500 of size 256 by op assert_less_equal_1/Assert/AssertGuard/pivot_f/_41 action_count 94195480111371 step 0 next 1133
    2021-08-09 03:52:38.110080: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695600 of size 256 by op assert_less_equal_1/Assert/AssertGuard/pivot_t/_42 action_count 94195480111372 step 0 next 474
    2021-08-09 03:52:38.110109: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695700 of size 256 by op gradient_tape/unet_depth4/two_conv_decoder0/batch_normalization_601/moments/scalar action_count 94195480111373
    step 0 next 224
    2021-08-09 03:52:38.110137: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695800 of size 256 by op assert_greater_equal_2/Assert/AssertGuard/pivot_f/_59 action_count 94195480111375 step 0 next 1522
    2021-08-09 03:52:38.110163: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695900 of size 256 by op assert_greater_equal_2/Assert/AssertGuard/pivot_t/_60 action_count 94195480111376 step 0 next 598
    2021-08-09 03:52:38.110190: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695a00 of size 256 by op assert_less_equal_2/Assert/AssertGuard/pivot_f/_69 action_count 94195480111377 step 0 next 1673
    2021-08-09 03:52:38.110226: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695b00 of size 256 by op assert_less_equal_2/Assert/AssertGuard/pivot_t/_70 action_count 94195480111378 step 0 next 1565
    2021-08-09 03:52:38.110257: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695c00 of size 256 by op assert_greater_equal_3/Assert/AssertGuard/pivot_f/_87 action_count 94195480111379 step 0 next 1719
    2021-08-09 03:52:38.110282: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695d00 of size 256 by op assert_greater_equal_3/Assert/AssertGuard/pivot_t/_88 action_count 94195480111380 step 0 next 1730
    2021-08-09 03:52:38.110309: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695e00 of size 256 by op assert_less_equal_3/Assert/AssertGuard/pivot_f/_97 action_count 94195480111381 step 0 next 1514
    2021-08-09 03:52:38.110337: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe695f00 of size 256 by op assert_less_equal_3/Assert/AssertGuard/pivot_t/_98 action_count 94195480111382 step 0 next 1671
    2021-08-09 03:52:38.110365: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696000 of size 256 by op gradient_tape/unet_depth4/two_conv_decoder0/batch_normalization_600/moments/scalar action_count 94195480111383
    step 0 next 1576
    2021-08-09 03:52:38.110403: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696100 of size 256 by op assert_greater_equal_4/Assert/AssertGuard/pivot_f/_115 action_count 94195480111384 step 0 next 1597
    2021-08-09 03:52:38.110431: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696200 of size 256 by op assert_greater_equal_4/Assert/AssertGuard/pivot_t/_116 action_count 94195480111385 step 0 next 1512
    2021-08-09 03:52:38.110458: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696300 of size 256 by op assert_less_equal_4/Assert/AssertGuard/pivot_f/_125 action_count 94195480111386 step 0 next 1558
    2021-08-09 03:52:38.110484: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696400 of size 256 by op assert_less_equal_4/Assert/AssertGuard/pivot_t/_126 action_count 94195480111387 step 0 next 1539
    2021-08-09 03:52:38.110511: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696500 of size 256 by op assert_greater_equal_5/Assert/AssertGuard/pivot_f/_143 action_count 94195480111388 step 0 next 1536
    2021-08-09 03:52:38.110545: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696600 of size 256 by op assert_greater_equal_5/Assert/AssertGuard/pivot_t/_144 action_count 94195480111389 step 0 next 1590
    2021-08-09 03:52:38.110583: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696700 of size 256 by op gradient_tape/unet_depth4/conv_transpose_decoder0/batch_normalization_599/moments/scalar action_count 941954801
    11390 step 0 next 1725
    2021-08-09 03:52:38.110613: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696800 of size 256 by op assert_less_equal_5/Assert/AssertGuard/pivot_f/_153 action_count 94195480111391 step 0 next 1676
    2021-08-09 03:52:38.110642: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696900 of size 256 by op assert_less_equal_5/Assert/AssertGuard/pivot_t/_154 action_count 94195480111392 step 0 next 1632
    2021-08-09 03:52:38.110669: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696a00 of size 256 by op assert_greater_equal_6/Assert/AssertGuard/pivot_f/_171 action_count 94195480111393 step 0 next 1532
    2021-08-09 03:52:38.110696: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696b00 of size 256 by op assert_greater_equal_6/Assert/AssertGuard/pivot_t/_172 action_count 94195480111394 step 0 next 1718
    2021-08-09 03:52:38.110722: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696c00 of size 256 by op assert_less_equal_6/Assert/AssertGuard/pivot_f/_181 action_count 94195480111395 step 0 next 427
    2021-08-09 03:52:38.110758: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696d00 of size 256 by op assert_less_equal_6/Assert/AssertGuard/pivot_t/_182 action_count 94195480111396 step 0 next 1553
    2021-08-09 03:52:38.110788: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696e00 of size 256 by op gradient_tape/unet_depth4/two_conv_decoder1/batch_normalization_598/moments/scalar action_count 94195480111397
    step 0 next 516
    2021-08-09 03:52:38.110814: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe696f00 of size 256 by op assert_greater_equal_7/Assert/AssertGuard/pivot_f/_199 action_count 94195480111398 step 0 next 1550
    2021-08-09 03:52:38.110841: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697000 of size 256 by op assert_greater_equal_7/Assert/AssertGuard/pivot_t/_200 action_count 94195480111399 step 0 next 129
    2021-08-09 03:52:38.110867: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697100 of size 256 by op assert_less_equal_7/Assert/AssertGuard/pivot_f/_209 action_count 94195480111400 step 0 next 369
    2021-08-09 03:52:38.110894: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697200 of size 256 by op assert_less_equal_7/Assert/AssertGuard/pivot_t/_210 action_count 94195480111401 step 0 next 1542
    2021-08-09 03:52:38.110920: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697300 of size 256 by op assert_greater_equal_8/Assert/AssertGuard/pivot_f/_227 action_count 94195480111402 step 0 next 433
    2021-08-09 03:52:38.110958: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697400 of size 256 by op assert_greater_equal_8/Assert/AssertGuard/pivot_t/_228 action_count 94195480111403 step 0 next 1678
    2021-08-09 03:52:38.110986: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697500 of size 256 by op gradient_tape/unet_depth4/two_conv_decoder1/batch_normalization_597/moments/scalar action_count 94195480111404
    step 0 next 241
    2021-08-09 03:52:38.111015: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697600 of size 256 by op assert_less_equal_8/Assert/AssertGuard/pivot_f/_237 action_count 94195480111405 step 0 next 1776
    2021-08-09 03:52:38.111052: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697700 of size 256 by op assert_less_equal_8/Assert/AssertGuard/pivot_t/_238 action_count 94195480111406 step 0 next 1568
    2021-08-09 03:52:38.111081: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697800 of size 256 by op assert_greater_equal_9/Assert/AssertGuard/pivot_f/_255 action_count 94195480111407 step 0 next 951
    2021-08-09 03:52:38.111109: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697900 of size 256 by op assert_greater_equal_9/Assert/AssertGuard/pivot_t/_256 action_count 94195480111408 step 0 next 1720
    2021-08-09 03:52:38.111135: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697a00 of size 256 by op assert_less_equal_9/Assert/AssertGuard/pivot_f/_265 action_count 94195480111409 step 0 next 538
    2021-08-09 03:52:38.111161: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697b00 of size 256 by op assert_less_equal_9/Assert/AssertGuard/pivot_t/_266 action_count 94195480111410 step 0 next 1287
    2021-08-09 03:52:38.111190: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697c00 of size 256 by op assert_greater_equal_10/Assert/AssertGuard/pivot_f/_283 action_count 94195480111411 step 0 next 549
    2021-08-09 03:52:38.111218: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697d00 of size 256 by op assert_greater_equal_10/Assert/AssertGuard/pivot_t/_284 action_count 94195480111412 step 0 next 605
    2021-08-09 03:52:38.111243: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697e00 of size 256 by op gradient_tape/unet_depth4/conv_transpose_decoder1/batch_normalization_596/moments/scalar action_count 941954801
    11413 step 0 next 1567
    2021-08-09 03:52:38.111283: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe697f00 of size 256 by op assert_less_equal_10/Assert/AssertGuard/pivot_f/_293 action_count 94195480111414 step 0 next 111
    2021-08-09 03:52:38.111311: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698000 of size 256 by op assert_less_equal_10/Assert/AssertGuard/pivot_t/_294 action_count 94195480111415 step 0 next 390
    2021-08-09 03:52:38.111340: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698100 of size 256 by op assert_greater_equal_11/Assert/AssertGuard/pivot_f/_311 action_count 94195480111416 step 0 next 907
    2021-08-09 03:52:38.111366: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698200 of size 256 by op assert_greater_equal_11/Assert/AssertGuard/pivot_t/_312 action_count 94195480111417 step 0 next 1209
    2021-08-09 03:52:38.111393: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698300 of size 256 by op assert_less_equal_11/Assert/AssertGuard/pivot_f/_321 action_count 94195480111418 step 0 next 289
    2021-08-09 03:52:38.111430: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698400 of size 256 by op assert_less_equal_11/Assert/AssertGuard/pivot_t/_322 action_count 94195480111419 step 0 next 1891
    2021-08-09 03:52:38.111458: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698500 of size 256 by op gradient_tape/unet_depth4/two_conv_decoder2/batch_normalization_595/moments/scalar action_count 94195480111420
    step 0 next 268
    2021-08-09 03:52:38.111485: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698600 of size 256 by op assert_greater_equal_12/Assert/AssertGuard/pivot_f/_339 action_count 94195480111421 step 0 next 118
    2021-08-09 03:52:38.111511: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698700 of size 256 by op assert_greater_equal_12/Assert/AssertGuard/pivot_t/_340 action_count 94195480111422 step 0 next 1145
    2021-08-09 03:52:38.111545: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698800 of size 256 by op assert_less_equal_12/Assert/AssertGuard/pivot_f/_349 action_count 94195480111423 step 0 next 893
    2021-08-09 03:52:38.111574: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698900 of size 256 by op assert_less_equal_12/Assert/AssertGuard/pivot_t/_350 action_count 94195480111424 step 0 next 1114
    2021-08-09 03:52:38.111603: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698a00 of size 256 by op assert_greater_equal_13/Assert/AssertGuard/pivot_f/_367 action_count 94195480111425 step 0 next 1753
    2021-08-09 03:52:38.111640: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698b00 of size 256 by op assert_greater_equal_13/Assert/AssertGuard/pivot_t/_368 action_count 94195480111426 step 0 next 1264
    2021-08-09 03:52:38.111667: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698c00 of size 256 by op gradient_tape/unet_depth4/two_conv_decoder2/batch_normalization_594/moments/scalar action_count 94195480111427
    step 0 next 475
    2021-08-09 03:52:38.111706: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698d00 of size 256 by op assert_less_equal_13/Assert/AssertGuard/pivot_f/_377 action_count 94195480111428 step 0 next 1060
    2021-08-09 03:52:38.111730: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698e00 of size 256 by op assert_less_equal_13/Assert/AssertGuard/pivot_t/_378 action_count 94195480111429 step 0 next 1700
    2021-08-09 03:52:38.111755: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe698f00 of size 256 by op assert_greater_equal_14/Assert/AssertGuard/pivot_f/_395 action_count 94195480111430 step 0 next 1755
    2021-08-09 03:52:38.111789: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699000 of size 256 by op assert_greater_equal_14/Assert/AssertGuard/pivot_t/_396 action_count 94195480111431 step 0 next 2062
    2021-08-09 03:52:38.111813: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699100 of size 256 by op assert_less_equal_14/Assert/AssertGuard/pivot_f/_405 action_count 94195480111432 step 0 next 1877
    2021-08-09 03:52:38.111838: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699200 of size 256 by op assert_less_equal_14/Assert/AssertGuard/pivot_t/_406 action_count 94195480111433 step 0 next 1596
    2021-08-09 03:52:38.111862: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699300 of size 256 by op gradient_tape/unet_depth4/conv_transpose_decoder2/batch_normalization_593/moments/scalar action_count 941954801
    11434 step 0 next 1140
    2021-08-09 03:52:38.111888: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699400 of size 256 by op assert_greater_equal_15/Assert/AssertGuard/pivot_f/_423 action_count 94195480111435 step 0 next 418
    2021-08-09 03:52:38.111913: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699500 of size 256 by op assert_greater_equal_15/Assert/AssertGuard/pivot_t/_424 action_count 94195480111436 step 0 next 2194
    2021-08-09 03:52:38.111947: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699600 of size 256 by op assert_less_equal_15/Assert/AssertGuard/pivot_f/_433 action_count 94195480111437 step 0 next 876
    2021-08-09 03:52:38.111975: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699700 of size 256 by op assert_less_equal_15/Assert/AssertGuard/pivot_t/_434 action_count 94195480111438 step 0 next 387
    2021-08-09 03:52:38.112003: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699800 of size 256 by op assert_greater_equal_16/Assert/AssertGuard/pivot_f/_451 action_count 94195480111439 step 0 next 2048
    2021-08-09 03:52:38.112030: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699900 of size 256 by op assert_greater_equal_16/Assert/AssertGuard/pivot_t/_452 action_count 94195480111440 step 0 next 1297
    2021-08-09 03:52:38.112060: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699a00 of size 256 by op gradient_tape/unet_depth4/two_conv_decoder3/batch_normalization_592/moments/scalar action_count 94195480111441
    step 0 next 1552
    2021-08-09 03:52:38.112095: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699b00 of size 256 by op assert_less_equal_16/Assert/AssertGuard/pivot_f/_461 action_count 94195480111442 step 0 next 1858
    2021-08-09 03:52:38.112124: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699c00 of size 256 by op assert_less_equal_16/Assert/AssertGuard/pivot_t/_462 action_count 94195480111443 step 0 next 1274
    2021-08-09 03:52:38.112150: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699d00 of size 256 by op assert_greater_equal_17/Assert/AssertGuard/pivot_f/_479 action_count 94195480111444 step 0 next 2097
    2021-08-09 03:52:38.112177: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699e00 of size 256 by op assert_greater_equal_17/Assert/AssertGuard/pivot_t/_480 action_count 94195480111445 step 0 next 55
    2021-08-09 03:52:38.112203: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe699f00 of size 256 by op assert_less_equal_17/Assert/AssertGuard/pivot_f/_489 action_count 94195480111446 step 0 next 1333
    2021-08-09 03:52:38.112230: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69a000 of size 256 by op assert_less_equal_17/Assert/AssertGuard/pivot_t/_490 action_count 94195480111447 step 0 next 1263
    2021-08-09 03:52:38.112256: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69a100 of size 256 by op assert_greater_equal_18/Assert/AssertGuard/pivot_f/_507 action_count 94195480111448 step 0 next 1277
    2021-08-09 03:52:38.112286: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69a200 of size 256 by op assert_greater_equal_18/Assert/AssertGuard/pivot_t/_508 action_count 94195480111449 step 0 next 1079
    2021-08-09 03:52:38.112314: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69a300 of size 256 by op gradient_tape/unet_depth4/two_conv_decoder3/batch_normalization_591/moments/scalar action_count 94195480111450
    step 0 next 1543
    2021-08-09 03:52:38.112339: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69a400 of size 256 by op assert_less_equal_18/Assert/AssertGuard/pivot_f/_517 action_count 94195480111451 step 0 next 1722
    2021-08-09 03:52:38.112365: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69a500 of size 256 by op assert_less_equal_18/Assert/AssertGuard/pivot_t/_518 action_count 94195480111452 step 0 next 2614
    2021-08-09 03:52:38.112403: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69a600 of size 256 by op assert_greater_equal_19/Assert/AssertGuard/pivot_f/_535 action_count 94195480111453 step 0 next 365
    2021-08-09 03:52:38.112429: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69a700 of size 256 by op assert_greater_equal_19/Assert/AssertGuard/pivot_t/_536 action_count 94195480111454 step 0 next 1068
    2021-08-09 03:52:38.112456: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69a800 of size 256 by op assert_less_equal_19/Assert/AssertGuard/pivot_f/_545 action_count 94195480111455 step 0 next 900
    2021-08-09 03:52:38.112481: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69a900 of size 256 by op assert_less_equal_19/Assert/AssertGuard/pivot_t/_546 action_count 94195480111456 step 0 next 802
    2021-08-09 03:52:38.112508: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69aa00 of size 256 by op gradient_tape/unet_depth4/conv_transpose_decoder3/batch_normalization_590/moments/scalar action_count 941954801
    11457 step 0 next 2922
    2021-08-09 03:52:38.112542: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69ab00 of size 256 by op assert_greater_equal_20/Assert/AssertGuard/pivot_f/_563 action_count 94195480111458 step 0 next 2994
    2021-08-09 03:52:38.112566: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69ac00 of size 256 by op assert_greater_equal_20/Assert/AssertGuard/pivot_t/_564 action_count 94195480111459 step 0 next 1443
    2021-08-09 03:52:38.112589: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69ad00 of size 256 by op assert_less_equal_20/Assert/AssertGuard/pivot_f/_573 action_count 94195480111460 step 0 next 3039
    2021-08-09 03:52:38.112612: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69ae00 of size 256 by op assert_less_equal_20/Assert/AssertGuard/pivot_t/_574 action_count 94195480111461 step 0 next 2814
    2021-08-09 03:52:38.112646: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69af00 of size 256 by op assert_greater_equal_21/Assert/AssertGuard/pivot_f/_591 action_count 94195480111462 step 0 next 2808
    2021-08-09 03:52:38.112673: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69b000 of size 256 by op assert_greater_equal_21/Assert/AssertGuard/pivot_t/_592 action_count 94195480111463 step 0 next 3093
    2021-08-09 03:52:38.112697: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69b100 of size 256 by op gradient_tape/unet_depth4/two_conv_center/batch_normalization_589/moments/scalar action_count 94195480111464 st
    ep 0 next 2882
    2021-08-09 03:52:38.112721: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69b200 of size 256 by op assert_less_equal_21/Assert/AssertGuard/pivot_f/_601 action_count 94195480111465 step 0 next 2963
    2021-08-09 03:52:38.112755: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69b300 of size 256 by op assert_less_equal_21/Assert/AssertGuard/pivot_t/_602 action_count 94195480111466 step 0 next 1412
    2021-08-09 03:52:38.112781: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69b400 of size 256 by op assert_greater_equal_22/Assert/AssertGuard/pivot_f/_619 action_count 94195480111467 step 0 next 2849
    2021-08-09 03:52:38.112805: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69b500 of size 256 by op assert_greater_equal_22/Assert/AssertGuard/pivot_t/_620 action_count 94195480111468 step 0 next 2795
    2021-08-09 03:52:38.112830: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69b600 of size 256 by op assert_less_equal_22/Assert/AssertGuard/pivot_f/_629 action_count 94195480111469 step 0 next 2932
    2021-08-09 03:52:38.112856: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69b700 of size 256 by op assert_less_equal_22/Assert/AssertGuard/pivot_t/_630 action_count 94195480111470 step 0 next 2770
    2021-08-09 03:52:38.112880: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69b800 of size 256 by op gradient_tape/unet_depth4/two_conv_center/batch_normalization_588/moments/scalar action_count 94195480111471 st
    ep 0 next 2243
    2021-08-09 03:52:38.112905: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69b900 of size 256 by op assert_greater_equal_23/Assert/AssertGuard/pivot_f/_647 action_count 94195480111472 step 0 next 2759
    2021-08-09 03:52:38.112929: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69ba00 of size 256 by op assert_greater_equal_23/Assert/AssertGuard/pivot_t/_648 action_count 94195480111473 step 0 next 2741
    2021-08-09 03:52:38.112965: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69bb00 of size 256 by op assert_less_equal_23/Assert/AssertGuard/pivot_f/_657 action_count 94195480111474 step 0 next 443
    2021-08-09 03:52:38.113001: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69bc00 of size 256 by op assert_less_equal_23/Assert/AssertGuard/pivot_t/_658 action_count 94195480111475 step 0 next 2897
    2021-08-09 03:52:38.113027: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69bd00 of size 256 by op assert_greater_equal_24/Assert/AssertGuard/pivot_f/_675 action_count 94195480111476 step 0 next 2857
    2021-08-09 03:52:38.113050: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69be00 of size 256 by op assert_greater_equal_24/Assert/AssertGuard/pivot_t/_676 action_count 94195480111477 step 0 next 3087
    2021-08-09 03:52:38.113073: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69bf00 of size 256 by op assert_less_equal_24/Assert/AssertGuard/pivot_f/_685 action_count 94195480111478 step 0 next 3055
    2021-08-09 03:52:38.113100: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69c000 of size 256 by op assert_less_equal_24/Assert/AssertGuard/pivot_t/_686 action_count 94195480111479 step 0 next 2949
    2021-08-09 03:52:38.113125: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69c100 of size 256 by op gradient_tape/unet_depth4/encode3/batch_normalization_587/moments/scalar action_count 94195480111480 step 0 nex
    t 2981
    2021-08-09 03:52:38.113159: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69c200 of size 256 by op assert_greater_equal_25/Assert/AssertGuard/pivot_f/_703 action_count 94195480111481 step 0 next 3083
    2021-08-09 03:52:38.113180: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69c300 of size 256 by op assert_greater_equal_25/Assert/AssertGuard/pivot_t/_704 action_count 94195480111482 step 0 next 3088
    2021-08-09 03:52:38.113203: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69c400 of size 256 by op assert_less_equal_25/Assert/AssertGuard/pivot_f/_713 action_count 94195480111483 step 0 next 2752
    2021-08-09 03:52:38.113226: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69c500 of size 256 by op assert_less_equal_25/Assert/AssertGuard/pivot_t/_714 action_count 94195480111484 step 0 next 3048
    2021-08-09 03:52:38.113253: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69c600 of size 256 by op assert_greater_equal_26/Assert/AssertGuard/pivot_f/_731 action_count 94195480111485 step 0 next 3002
    2021-08-09 03:52:38.113285: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69c700 of size 256 by op assert_greater_equal_26/Assert/AssertGuard/pivot_t/_732 action_count 94195480111486 step 0 next 349
    2021-08-09 03:52:38.113310: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69c800 of size 256 by op gradient_tape/unet_depth4/encode3/batch_normalization_586/moments/scalar action_count 94195480111487 step 0 nex
    t 2859
    2021-08-09 03:52:38.113334: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69c900 of size 256 by op assert_less_equal_26/Assert/AssertGuard/pivot_f/_741 action_count 94195480111488 step 0 next 2696
    2021-08-09 03:52:38.113358: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69ca00 of size 256 by op assert_less_equal_26/Assert/AssertGuard/pivot_t/_742 action_count 94195480111489 step 0 next 2629
    2021-08-09 03:52:38.113385: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69cb00 of size 256 by op assert_greater_equal_27/Assert/AssertGuard/pivot_f/_759 action_count 94195480111490 step 0 next 3008
    2021-08-09 03:52:38.113405: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69cc00 of size 256 by op assert_greater_equal_27/Assert/AssertGuard/pivot_t/_760 action_count 94195480111491 step 0 next 2093
    2021-08-09 03:52:38.113428: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69cd00 of size 256 by op assert_less_equal_27/Assert/AssertGuard/pivot_f/_769 action_count 94195480111492 step 0 next 2978
    2021-08-09 03:52:38.113459: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69ce00 of size 256 by op assert_less_equal_27/Assert/AssertGuard/pivot_t/_770 action_count 94195480111493 step 0 next 3074
    2021-08-09 03:52:38.113485: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69cf00 of size 256 by op assert_greater_equal_28/Assert/AssertGuard/pivot_f/_787 action_count 94195480111494 step 0 next 3075
    2021-08-09 03:52:38.113509: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69d000 of size 256 by op assert_greater_equal_28/Assert/AssertGuard/pivot_t/_788 action_count 94195480111495 step 0 next 3018
    2021-08-09 03:52:38.113550: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69d100 of size 256 by op gradient_tape/unet_depth4/encode2/batch_normalization_585/moments/scalar action_count 94195480111496 step 0 nex
    t 2844
    2021-08-09 03:52:38.113577: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69d200 of size 256 by op assert_less_equal_28/Assert/AssertGuard/pivot_f/_797 action_count 94195480111497 step 0 next 1422
    2021-08-09 03:52:38.113601: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69d300 of size 256 by op assert_less_equal_28/Assert/AssertGuard/pivot_t/_798 action_count 94195480111498 step 0 next 3025
    2021-08-09 03:52:38.113625: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69d400 of size 256 by op assert_greater_equal_29/Assert/AssertGuard/pivot_f/_815 action_count 94195480111499 step 0 next 3003
    2021-08-09 03:52:38.113649: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69d500 of size 256 by op assert_greater_equal_29/Assert/AssertGuard/pivot_t/_816 action_count 94195480111500 step 0 next 2735
    2021-08-09 03:52:38.113674: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69d600 of size 256 by op assert_less_equal_29/Assert/AssertGuard/pivot_f/_825 action_count 94195480111501 step 0 next 2969
    2021-08-09 03:52:38.113706: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69d700 of size 256 by op assert_less_equal_29/Assert/AssertGuard/pivot_t/_826 action_count 94195480111502 step 0 next 2940
    2021-08-09 03:52:38.113731: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69d800 of size 256 by op assert_greater_equal_30/Assert/AssertGuard/pivot_f/_843 action_count 94195480111503 step 0 next 2957
    2021-08-09 03:52:38.113756: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69d900 of size 256 by op assert_greater_equal_30/Assert/AssertGuard/pivot_t/_844 action_count 94195480111504 step 0 next 2811
    2021-08-09 03:52:38.113780: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69da00 of size 256 by op gradient_tape/unet_depth4/encode2/batch_normalization_584/moments/scalar action_count 94195480111505 step 0 nex
    t 3012
    2021-08-09 03:52:38.113805: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69db00 of size 256 by op assert_less_equal_30/Assert/AssertGuard/pivot_f/_853 action_count 94195480111506 step 0 next 2782
    2021-08-09 03:52:38.113828: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69dc00 of size 256 by op assert_less_equal_30/Assert/AssertGuard/pivot_t/_854 action_count 94195480111507 step 0 next 2816
    2021-08-09 03:52:38.113851: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69dd00 of size 256 by op assert_greater_equal_31/Assert/AssertGuard/pivot_f/_871 action_count 94195480111508 step 0 next 1605
    2021-08-09 03:52:38.113876: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69de00 of size 256 by op assert_greater_equal_31/Assert/AssertGuard/pivot_t/_872 action_count 94195480111509 step 0 next 2740
    2021-08-09 03:52:38.113900: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69df00 of size 256 by op assert_less_equal_31/Assert/AssertGuard/pivot_f/_881 action_count 94195480111510 step 0 next 2942
    2021-08-09 03:52:38.113923: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69e000 of size 256 by op assert_less_equal_31/Assert/AssertGuard/pivot_t/_882 action_count 94195480111511 step 0 next 2905
    2021-08-09 03:52:38.113957: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69e100 of size 256 by op assert_greater_equal_32/Assert/AssertGuard/pivot_f/_899 action_count 94195480111512 step 0 next 2806
    2021-08-09 03:52:38.113981: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69e200 of size 256 by op assert_greater_equal_32/Assert/AssertGuard/pivot_t/_900 action_count 94195480111513 step 0 next 2987
    2021-08-09 03:52:38.114006: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69e300 of size 256 by op gradient_tape/unet_depth4/encode1/batch_normalization_583/moments/scalar action_count 94195480111514 step 0 nex
    t 2968
    2021-08-09 03:52:38.114031: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69e400 of size 256 by op assert_less_equal_32/Assert/AssertGuard/pivot_f/_909 action_count 94195480111515 step 0 next 3052
    2021-08-09 03:52:38.114055: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69e500 of size 256 by op assert_less_equal_32/Assert/AssertGuard/pivot_t/_910 action_count 94195480111516 step 0 next 3020
    2021-08-09 03:52:38.114080: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69e600 of size 256 by op gradient_tape/unet_depth4/encode1/batch_normalization_582/moments/scalar action_count 94195480111517 step 0 nex
    t 2773
    2021-08-09 03:52:38.114105: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69e700 of size 256 by op gradient_tape/unet_depth4/encode0/batch_normalization_581/moments/scalar action_count 94195480111518 step 0 nex
    t 2792
    2021-08-09 03:52:38.114141: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69e800 of size 256 by op gradient_tape/unet_depth4/encode0/batch_normalization_580/moments/scalar action_count 94195480111519 step 0 nex
    t 2842
    2021-08-09 03:52:38.114166: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69e900 of size 256 by op Adam/Adam/Const action_count 94195480111520 step 0 next 3023
    2021-08-09 03:52:38.114200: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe69ea00 of size 1441792 by op gradient_tape/binary_ce_dice/logistic_loss/sub/Neg/_0__cf__4291 action_count 94195480111521 step 0 next 279
    8
    2021-08-09 03:52:38.114227: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe7fea00 of size 1441792 by op gradient_tape/binary_ce_dice/weighted_loss/Tile_1/_1__cf__4292 action_count 94195480111522 step 0 next 2948
    2021-08-09 03:52:38.114252: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3afe95ea00 of size 157947392 by op Tile_90 action_count 94195480111525 step 13755587135625806913 next 18446744073709551615
    2021-08-09 03:52:38.114277: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Next region of size 2147483648
    2021-08-09 03:52:38.114304: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3b96000000 of size 340262912 by op Adam/gradients/AddN_3/tmp_var_zeros action_count 94195480111374 step 0 next 1663
    2021-08-09 03:52:38.114340: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3baa480000 of size 170131456 by op unet_depth4/encode0/conv1d_506/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 94195480111553
     step 13755587135625806913 next 2699
    2021-08-09 03:52:38.114370: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3bb46c0000 of size 512 by op unet_depth4/encode0/batch_normalization_580/moments/variance action_count 94195480111559 step 13755587135625806
    913 next 3044
    2021-08-09 03:52:38.114396: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3bb46c0200 of size 170131456 by op gradient_tape/unet_depth4/encode0/conv1d_507/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOpti
    mizer action_count 94195480111571 step 13755587135625806913 next 2506
    2021-08-09 03:52:38.114424: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3bbe900200 of size 512 by op unet_depth4/encode0/batch_normalization_581/moments/mean action_count 94195480111593 step 13755587135625806913
    next 2970
    2021-08-09 03:52:38.114457: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3bbe900400 of size 512 by op unet_depth4/encode0/batch_normalization_581/moments/variance action_count 94195480111597 step 13755587135625806
    913 next 2492
    2021-08-09 03:52:38.114485: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3bbe900600 of size 512 by op unet_depth4/encode0/batch_normalization_581/batchnorm/mul action_count 94195480111605 step 13755587135625806913
     next 3019
    2021-08-09 03:52:38.114512: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3bbe900800 of size 170129920 by op unet_depth4/mp_encode0/MaxPool action_count 94195480111610 step 13755587135625806913 next 2951
    2021-08-09 03:52:38.114545: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3bc8b40200 of size 170131456 by op unet_depth4/encode0/conv1d_507/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 94195480111591
     step 13755587135625806913 next 3090
    2021-08-09 03:52:38.114581: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3bd2d80200 of size 170131456 by op unet_depth4/encode0/batch_normalization_581/batchnorm/mul_1 action_count 94195480111606 step 137555871356
    25806913 next 2838
    2021-08-09 03:52:38.114610: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3bdcfc0200 of size 170131456 by op gradient_tape/unet_depth4/mp_encode0/MaxPool/MaxPoolGrad-0-TransposeNHWCToNCHW-LayoutOptimizer action_cou
    nt 94195480111609 step 13755587135625806913 next 2829
    2021-08-09 03:52:38.114638: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7200200 of size 1024 by op unet_depth4/encode1/batch_normalization_582/moments/mean action_count 94195480111635 step 13755587135625806913
     next 1051
    2021-08-09 03:52:38.114663: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7200600 of size 1024 by op unet_depth4/encode1/batch_normalization_582/moments/variance action_count 94195480111639 step 1375558713562580
    6913 next 2989
    2021-08-09 03:52:38.114696: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7200a00 of size 1024 by op unet_depth4/encode1/batch_normalization_582/batchnorm/mul action_count 94195480111647 step 1375558713562580691
    3 next 3040
    2021-08-09 03:52:38.114724: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7200e00 of size 1024 by op unet_depth4/encode1/batch_normalization_583/moments/mean action_count 94195480111673 step 13755587135625806913
     next 2344
    2021-08-09 03:52:38.114751: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7201200 of size 1024 by op unet_depth4/encode1/batch_normalization_583/moments/variance action_count 94195480111677 step 1375558713562580
    6913 next 2890
    2021-08-09 03:52:38.114778: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7201600 of size 1024 by op unet_depth4/encode1/batch_normalization_583/batchnorm/mul action_count 94195480111685 step 1375558713562580691
    3 next 2930
    2021-08-09 03:52:38.114802: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7201a00 of size 2048 by op unet_depth4/encode2/batch_normalization_584/moments/mean action_count 94195480111715 step 13755587135625806913
     next 3034
    2021-08-09 03:52:38.114836: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7202200 of size 2048 by op unet_depth4/encode2/batch_normalization_584/moments/variance action_count 94195480111719 step 1375558713562580
    6913 next 2704
    2021-08-09 03:52:38.114862: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7202a00 of size 2048 by op unet_depth4/encode2/batch_normalization_584/batchnorm/mul action_count 94195480111727 step 1375558713562580691
    3 next 3066
    2021-08-09 03:52:38.114888: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7203200 of size 2048 by op unet_depth4/encode2/batch_normalization_585/moments/mean action_count 94195480111753 step 13755587135625806913
     next 1476
    2021-08-09 03:52:38.114914: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7203a00 of size 2048 by op unet_depth4/encode2/batch_normalization_585/moments/variance action_count 94195480111757 step 1375558713562580
    6913 next 3091
    2021-08-09 03:52:38.114942: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7204200 of size 2048 by op unet_depth4/encode2/batch_normalization_585/batchnorm/mul action_count 94195480111765 step 1375558713562580691
    3 next 2729
    2021-08-09 03:52:38.114976: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7204a00 of size 2048 by op unet_depth4/encode3/batch_normalization_586/moments/mean action_count 94195480111795 step 13755587135625806913
     next 3041
    2021-08-09 03:52:38.115002: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7205200 of size 2048 by op unet_depth4/encode3/batch_normalization_586/moments/variance action_count 94195480111797 step 1375558713562580
    6913 next 3065
    2021-08-09 03:52:38.115027: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7205a00 of size 2048 by op unet_depth4/encode3/batch_normalization_586/batchnorm/mul action_count 94195480111803 step 1375558713562580691
    3 next 2971
    2021-08-09 03:52:38.115052: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7206200 of size 2048 by op unet_depth4/encode3/batch_normalization_587/moments/mean action_count 94195480111829 step 13755587135625806913
     next 1680
    2021-08-09 03:52:38.115076: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7206a00 of size 2048 by op unet_depth4/encode3/batch_normalization_587/moments/variance action_count 94195480111831 step 1375558713562580
    6913 next 2880
    2021-08-09 03:52:38.115112: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7207200 of size 2048 by op unet_depth4/encode3/batch_normalization_587/batchnorm/mul action_count 94195480111837 step 1375558713562580691
    3 next 2712
    2021-08-09 03:52:38.115140: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3be7207a00 of size 85035008 by op unet_depth4/mp_encode3/MaxPool action_count 94195480111842 step 13755587135625806913 next 2763
    2021-08-09 03:52:38.115167: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3bec320200 of size 85065728 by op gradient_tape/unet_depth4/encode1/conv1d_508/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptim
    izer action_count 94195480111612 step 13755587135625806913 next 2765
    2021-08-09 03:52:38.115195: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3bf1440200 of size 170131456 by op unet_depth4/encode1/batch_normalization_582/batchnorm/mul_1 action_count 94195480111648 step 137555871356
    25806913 next 3086
    2021-08-09 03:52:38.115220: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3bfb680200 of size 170131456 by op unet_depth4/encode1/conv1d_508/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 94195480111633
     step 13755587135625806913 next 2790
    2021-08-09 03:52:38.115257: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3c058c0200 of size 276037120 by op gradient_tape/unet_depth4/encode1/conv1d_509/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOpti
    mizer action_count 94195480111651 step 13755587135625806913 next 18446744073709551615
    2021-08-09 03:52:38.115285: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Next region of size 4294967296
    2021-08-09 03:52:38.115309: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3c36000000 of size 170131456 by op unet_depth4/encode1/batch_normalization_583/batchnorm/mul_1 action_count 94195480111686 step 137555871356
    25806913 next 3009
    2021-08-09 03:52:38.115334: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3c40240000 of size 170131456 by op unet_depth4/encode1/conv1d_509/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 94195480111671
     step 13755587135625806913 next 2924
    2021-08-09 03:52:38.115360: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3c4a480000 of size 170131456 by op gradient_tape/unet_depth4/mp_encode1/MaxPool/MaxPoolGrad-0-TransposeNHWCToNCHW-LayoutOptimizer action_cou
    nt 94195480111689 step 13755587135625806913 next 3072
    2021-08-09 03:52:38.115388: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3c546c0000 of size 85065728 by op unet_depth4/mp_encode1/MaxPool action_count 94195480111690 step 13755587135625806913 next 2768
    2021-08-09 03:52:38.115423: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3c597e0000 of size 85065728 by op unet_depth4/mp_encode2/MaxPool action_count 94195480111770 step 13755587135625806913 next 737
    2021-08-09 03:52:38.115450: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3c5e900000 of size 85065728 by op gradient_tape/unet_depth4/encode2/conv1d_510/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptim
    izer action_count 94195480111692 step 13755587135625806913 next 3006
    2021-08-09 03:52:38.115479: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3c63a20000 of size 170131456 by op unet_depth4/encode2/batch_normalization_584/batchnorm/mul_1 action_count 94195480111728 step 137555871356
    25806913 next 3013
    2021-08-09 03:52:38.115504: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3c6dc60000 of size 170131456 by op unet_depth4/encode2/conv1d_510/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 94195480111713
     step 13755587135625806913 next 3035
    2021-08-09 03:52:38.115548: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3c77ea0000 of size 170131456 by op gradient_tape/unet_depth4/encode2/conv1d_511/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOpti
    mizer action_count 94195480111731 step 13755587135625806913 next 2953
    2021-08-09 03:52:38.115576: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3c820e0000 of size 170131456 by op unet_depth4/encode2/batch_normalization_585/batchnorm/mul_1 action_count 94195480111766 step 137555871356
    25806913 next 2995
    2021-08-09 03:52:38.115602: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3c8c320000 of size 170131456 by op unet_depth4/encode2/conv1d_511/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 94195480111751
     step 13755587135625806913 next 2966
    2021-08-09 03:52:38.115628: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3c96560000 of size 170131456 by op gradient_tape/unet_depth4/mp_encode2/MaxPool/MaxPoolGrad-0-TransposeNHWCToNCHW-LayoutOptimizer action_cou
    nt 94195480111769 step 13755587135625806913 next 2819
    2021-08-09 03:52:38.115655: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07a0000 of size 4096 by op unet_depth4/two_conv_center/batch_normalization_588/moments/mean action_count 94195480111867 step 137555871356
    25806913 next 2956
    2021-08-09 03:52:38.115680: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07a1000 of size 4096 by op unet_depth4/two_conv_center/batch_normalization_588/moments/variance action_count 94195480111869 step 13755587
    135625806913 next 2939
    2021-08-09 03:52:38.115705: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07a2000 of size 4096 by op unet_depth4/two_conv_center/batch_normalization_588/batchnorm/mul action_count 94195480111875 step 13755587135
    625806913 next 2727
    2021-08-09 03:52:38.115730: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07a3000 of size 4096 by op unet_depth4/two_conv_center/batch_normalization_589/moments/mean action_count 94195480111901 step 137555871356
    25806913 next 3085
    2021-08-09 03:52:38.115767: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07a4000 of size 4096 by op unet_depth4/two_conv_center/batch_normalization_589/moments/variance action_count 94195480111903 step 13755587
    135625806913 next 3046
    2021-08-09 03:52:38.115793: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07a5000 of size 4096 by op unet_depth4/two_conv_center/batch_normalization_589/batchnorm/mul action_count 94195480111909 step 13755587135
    625806913 next 2783
    2021-08-09 03:52:38.115819: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07a6000 of size 2048 by op unet_depth4/conv_transpose_decoder3/batch_normalization_590/moments/mean action_count 94195480111923 step 1375
    5587135625806913 next 2980
    2021-08-09 03:52:38.115844: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07a6800 of size 2048 by op unet_depth4/conv_transpose_decoder3/batch_normalization_590/batchnorm/mul action_count 94195480111931 step 137
    55587135625806913 next 2935
    2021-08-09 03:52:38.115880: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07a7000 of size 2048 by op unet_depth4/conv_transpose_decoder3/batch_normalization_590/moments/variance action_count 94195480111926 step
    13755587135625806913 next 2919
    2021-08-09 03:52:38.115908: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07a7800 of size 2048 by op unet_depth4/two_conv_decoder3/batch_normalization_591/moments/mean action_count 94195480111959 step 1375558713
    5625806913 next 3064
    2021-08-09 03:52:38.115934: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07a8000 of size 2048 by op unet_depth4/two_conv_decoder3/batch_normalization_591/batchnorm/mul action_count 94195480111967 step 137555871
    35625806913 next 2746
    2021-08-09 03:52:38.115962: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07a8800 of size 2048 by op unet_depth4/two_conv_decoder3/batch_normalization_591/moments/variance action_count 94195480111962 step 137555
    87135625806913 next 2778
    2021-08-09 03:52:38.115988: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07a9000 of size 2048 by op unet_depth4/two_conv_decoder3/batch_normalization_592/moments/mean action_count 94195480111979 step 1375558713
    5625806913 next 3014
    2021-08-09 03:52:38.116011: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07a9800 of size 2048 by op unet_depth4/two_conv_decoder3/batch_normalization_592/batchnorm/add action_count 94195480111985 step 137555871
    35625806913 next 3005
    2021-08-09 03:52:38.116045: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07aa000 of size 2048 by op unet_depth4/conv_transpose_decoder2/batch_normalization_593/moments/mean action_count 94195480112001 step 1375
    5587135625806913 next 3038
    2021-08-09 03:52:38.116073: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07aa800 of size 2048 by op unet_depth4/two_conv_decoder3/batch_normalization_592/batchnorm/mul action_count 94195480111986 step 137555871
    35625806913 next 2885
    2021-08-09 03:52:38.116106: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07ab000 of size 2048 by op unet_depth4/conv_transpose_decoder2/batch_normalization_593/batchnorm/mul action_count 94195480112009 step 137
    55587135625806913 next 2738
    2021-08-09 03:52:38.116135: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07ab800 of size 2048 by op unet_depth4/conv_transpose_decoder2/batch_normalization_593/moments/variance action_count 94195480112004 step
    13755587135625806913 next 2695
    2021-08-09 03:52:38.116161: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07ac000 of size 2048 by op unet_depth4/two_conv_decoder2/batch_normalization_594/moments/mean action_count 94195480112037 step 1375558713
    5625806913 next 2869
    2021-08-09 03:52:38.116186: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07ac800 of size 2048 by op unet_depth4/two_conv_decoder2/batch_normalization_594/batchnorm/mul action_count 94195480112045 step 137555871
    35625806913 next 3054
    2021-08-09 03:52:38.116210: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07ad000 of size 2048 by op unet_depth4/two_conv_decoder2/batch_normalization_594/moments/variance action_count 94195480112040 step 137555
    87135625806913 next 2959
    2021-08-09 03:52:38.116237: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07ad800 of size 2048 by op unet_depth4/two_conv_decoder2/batch_normalization_595/moments/mean action_count 94195480112071 step 1375558713
    5625806913 next 2760
    2021-08-09 03:52:38.116260: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07ae000 of size 2048 by op unet_depth4/two_conv_decoder2/batch_normalization_595/batchnorm/add action_count 94195480112077 step 137555871
    35625806913 next 3031
    2021-08-09 03:52:38.116296: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07ae800 of size 2048 by op unet_depth4/conv_transpose_decoder1/batch_normalization_596/moments/mean action_count 94195480112093 step 1375
    5587135625806913 next 2972
    2021-08-09 03:52:38.116323: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07af000 of size 2048 by op unet_depth4/two_conv_decoder2/batch_normalization_595/batchnorm/mul action_count 94195480112078 step 137555871
    35625806913 next 2903
    2021-08-09 03:52:38.116360: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07af800 of size 2048 by op unet_depth4/conv_transpose_decoder1/batch_normalization_596/batchnorm/mul action_count 94195480112105 step 137
    55587135625806913 next 2788
    2021-08-09 03:52:38.116387: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07b0000 of size 2048 by op unet_depth4/conv_transpose_decoder1/batch_normalization_596/moments/variance action_count 94195480112098 step
    13755587135625806913 next 3004
    2021-08-09 03:52:38.116413: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07b0800 of size 2048 by op unet_depth4/two_conv_decoder1/batch_normalization_597/moments/mean action_count 94195480112133 step 1375558713
    5625806913 next 2706
    2021-08-09 03:52:38.116438: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07b1000 of size 2048 by op unet_depth4/two_conv_decoder1/batch_normalization_597/batchnorm/mul action_count 94195480112145 step 137555871
    35625806913 next 3061
    2021-08-09 03:52:38.116463: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07b1800 of size 2048 by op unet_depth4/two_conv_decoder1/batch_normalization_597/moments/variance action_count 94195480112138 step 137555
    87135625806913 next 2962
    2021-08-09 03:52:38.116499: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07b2000 of size 2048 by op unet_depth4/two_conv_decoder1/batch_normalization_598/moments/mean action_count 94195480112171 step 1375558713
    5625806913 next 2889
    2021-08-09 03:52:38.116533: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07b2800 of size 2048 by op unet_depth4/two_conv_decoder1/batch_normalization_598/batchnorm/add action_count 94195480112181 step 137555871
    35625806913 next 2926
    2021-08-09 03:52:38.116561: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07b3000 of size 1024 by op unet_depth4/conv_transpose_decoder0/batch_normalization_599/moments/mean action_count 94195480112197 step 1375
    5587135625806913 next 3043
    2021-08-09 03:52:38.116587: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07b3400 of size 1024 by op unet_depth4/conv_transpose_decoder0/batch_normalization_599/batchnorm/mul action_count 94195480112209 step 137
    55587135625806913 next 2955
    2021-08-09 03:52:38.116622: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07b3800 of size 2048 by op unet_depth4/two_conv_decoder1/batch_normalization_598/batchnorm/mul action_count 94195480112182 step 137555871
    35625806913 next 3028
    2021-08-09 03:52:38.116648: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca07b4000 of size 1024 by op unet_depth4/conv_transpose_decoder0/batch_normalization_599/moments/variance action_count 94195480112202 step
    13755587135625806913 next 3081
    2021-08-09 03:52:38.116675: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free  at 2b3ca07b4400 of size 84982784 by op UNUSED action_count 94195480112212 step 13755587135625806913 next 1631
    2021-08-09 03:52:38.116700: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ca58c0000 of size 85065728 by op gradient_tape/unet_depth4/encode3/conv1d_512/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptim
    izer action_count 94195480111772 step 13755587135625806913 next 2950
    2021-08-09 03:52:38.116727: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3caa9e0000 of size 92274688 by op unet_depth4/encode3/batch_normalization_586/batchnorm/mul_1 action_count 94195480111804 step 1375558713562
    5806913 next 3082
    2021-08-09 03:52:38.116762: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3cb01e0000 of size 92274688 by op unet_depth4/encode3/conv1d_512/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 94195480111793
    step 13755587135625806913 next 412
    2021-08-09 03:52:38.116790: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3cb59e0000 of size 92274688 by op gradient_tape/unet_depth4/encode3/conv1d_513/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptim
    izer action_count 94195480111807 step 13755587135625806913 next 3053
    2021-08-09 03:52:38.116817: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3cbb1e0000 of size 92274688 by op unet_depth4/encode3/batch_normalization_587/batchnorm/mul_1 action_count 94195480111838 step 1375558713562
    5806913 next 2988
    2021-08-09 03:52:38.116842: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3cc09e0000 of size 92274688 by op unet_depth4/encode3/conv1d_513/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 94195480111827
    step 13755587135625806913 next 2982
    2021-08-09 03:52:38.116869: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3cc61e0000 of size 92274688 by op gradient_tape/unet_depth4/mp_encode3/MaxPool/MaxPoolGrad-0-TransposeNHWCToNCHW-LayoutOptimizer action_coun
    t 94195480111841 step 13755587135625806913 next 2944
    2021-08-09 03:52:38.116905: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ccb9e0000 of size 46137344 by op gradient_tape/unet_depth4/two_conv_center/conv1d_514/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-Lay
    outOptimizer action_count 94195480111844 step 13755587135625806913 next 2964
    2021-08-09 03:52:38.116933: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3cce5e0000 of size 92274688 by op unet_depth4/two_conv_center/batch_normalization_588/batchnorm/mul_1 action_count 94195480111876 step 13755
    587135625806913 next 2991
    2021-08-09 03:52:38.116960: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3cd3de0000 of size 92274688 by op unet_depth4/two_conv_center/conv1d_514/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 9419548
    0111865 step 13755587135625806913 next 2719
    2021-08-09 03:52:38.116986: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3cd95e0000 of size 92274688 by op gradient_tape/unet_depth4/two_conv_center/conv1d_515/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-Lay
    outOptimizer action_count 94195480111879 step 13755587135625806913 next 2694
    2021-08-09 03:52:38.117012: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3cdede0000 of size 92274688 by op unet_depth4/two_conv_center/batch_normalization_589/batchnorm/mul_1 action_count 94195480111910 step 13755
    587135625806913 next 770
    2021-08-09 03:52:38.117049: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ce45e0000 of size 92274688 by op unet_depth4/two_conv_center/conv1d_515/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 9419548
    0111899 step 13755587135625806913 next 2979
    2021-08-09 03:52:38.117077: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ce9de0000 of size 92274688 by op gradient_tape/unet_depth4/conv_transpose_decoder3/conv1d_transpose_104/conv1d_transpose/Conv2DBackpropFilt
    er-2-TransposeNHWCToNCHW-LayoutOptimizer action_count 94195480111913 step 13755587135625806913 next 2999
    2021-08-09 03:52:38.117104: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3cef5e0000 of size 92274688 by op unet_depth4/conv_transpose_decoder3/conv1d_transpose_104/conv1d_transpose-0-0-TransposeNCHWToNHWC-LayoutOp
    timizer action_count 94195480111921 step 13755587135625806913 next 2993
    2021-08-09 03:52:38.117131: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3cf4de0000 of size 92274688 by op unet_depth4/conv_transpose_decoder3/batch_normalization_590/batchnorm/mul_1 action_count 94195480111933 st
    ep 13755587135625806913 next 3071
    2021-08-09 03:52:38.117167: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3cfa5e0000 of size 92274688 by op unet_depth4/two_conv_decoder3/batch_normalization_591/batchnorm/mul_1 action_count 94195480111969 step 137
    55587135625806913 next 3001
    2021-08-09 03:52:38.117195: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3cffde0000 of size 92274688 by op unet_depth4/two_conv_decoder3/conv1d_516/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 94195
    480111957 step 13755587135625806913 next 2908
    2021-08-09 03:52:38.117223: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3d055e0000 of size 184549376 by op gradient_tape/unet_depth4/two_conv_decoder3/conv1d_516/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-
    LayoutOptimizer action_count 94195480111936 step 13755587135625806913 next 2977
    2021-08-09 03:52:38.117251: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3d105e0000 of size 92274688 by op gradient_tape/unet_depth4/two_conv_decoder3/conv1d_517/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-L
    ayoutOptimizer action_count 94195480111971 step 13755587135625806913 next 2736
    2021-08-09 03:52:38.117278: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3d15de0000 of size 92274688 by op unet_depth4/two_conv_decoder3/batch_normalization_592/batchnorm/mul_1 action_count 94195480111989 step 137
    55587135625806913 next 2910
    2021-08-09 03:52:38.117304: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3d1b5e0000 of size 92274688 by op unet_depth4/two_conv_decoder3/conv1d_517/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 94195
    480111977 step 13755587135625806913 next 2685
    2021-08-09 03:52:38.117331: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3d20de0000 of size 92274688 by op gradient_tape/unet_depth4/conv_transpose_decoder2/conv1d_transpose_105/conv1d_transpose/Conv2DBackpropFilt
    er-2-TransposeNHWCToNCHW-LayoutOptimizer action_count 94195480111991 step 13755587135625806913 next 3073
    2021-08-09 03:52:38.117358: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3d265e0000 of size 262275072 by op unet_depth4/conv_transpose_decoder2/conv1d_transpose_105/conv1d_transpose-0-0-TransposeNCHWToNHWC-LayoutO
    ptimizer action_count 94195480111999 step 13755587135625806913 next 18446744073709551615
    2021-08-09 03:52:38.117392: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Next region of size 6764888064
    2021-08-09 03:52:38.117420: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3d70000000 of size 184549376 by op unet_depth4/conv_transpose_decoder2/batch_normalization_593/batchnorm/mul_1 action_count 94195480112011 s
    tep 13755587135625806913 next 3032
    2021-08-09 03:52:38.117447: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3d7b000000 of size 184549376 by op unet_depth4/two_conv_decoder2/batch_normalization_594/batchnorm/mul_1 action_count 94195480112047 step 13
    755587135625806913 next 2626
    2021-08-09 03:52:38.117473: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free  at 2b3d86000000 of size 170131456 by op UNUSED action_count 94195421079702 step 17448270550709606267 next 2997
    2021-08-09 03:52:38.117499: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3d90240000 of size 354680832 by op gradient_tape/unet_depth4/two_conv_decoder2/conv1d_518/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-
    LayoutOptimizer action_count 94195480112014 step 13755587135625806913 next 2883
    2021-08-09 03:52:38.117543: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3da5480000 of size 184549376 by op unet_depth4/two_conv_decoder2/conv1d_518/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 9419
    5480112035 step 13755587135625806913 next 3010
    2021-08-09 03:52:38.117571: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3db0480000 of size 184549376 by op gradient_tape/unet_depth4/two_conv_decoder2/conv1d_519/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-
    LayoutOptimizer action_count 94195480112049 step 13755587135625806913 next 2943
    2021-08-09 03:52:38.117597: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3dbb480000 of size 184549376 by op unet_depth4/two_conv_decoder2/batch_normalization_595/batchnorm/mul_1 action_count 94195480112081 step 13
    755587135625806913 next 2973
    2021-08-09 03:52:38.117624: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3dc6480000 of size 184549376 by op unet_depth4/two_conv_decoder2/conv1d_519/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 9419
    5480112069 step 13755587135625806913 next 2984
    2021-08-09 03:52:38.117650: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3dd1480000 of size 184549376 by op gradient_tape/unet_depth4/conv_transpose_decoder1/conv1d_transpose_106/conv1d_transpose/Conv2DBackpropFil
    ter-2-TransposeNHWCToNCHW-LayoutOptimizer action_count 94195480112083 step 13755587135625806913 next 2946
    2021-08-09 03:52:38.117677: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ddc480000 of size 340262912 by op unet_depth4/conv_transpose_decoder1/conv1d_transpose_106/conv1d_transpose-0-0-TransposeNCHWToNHWC-LayoutO
    ptimizer action_count 94195480112091 step 13755587135625806913 next 2002
    2021-08-09 03:52:38.117713: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3df0900000 of size 340262912 by op unet_depth4/conv_transpose_decoder1/batch_normalization_596/batchnorm/mul_1 action_count 94195480112107 s
    tep 13755587135625806913 next 3036
    2021-08-09 03:52:38.117741: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3e04d80000 of size 340262912 by op unet_depth4/two_conv_decoder1/batch_normalization_597/batchnorm/mul_1 action_count 94195480112147 step 13
    755587135625806913 next 3062
    2021-08-09 03:52:38.117777: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free  at 2b3e19200000 of size 170131456 by op UNUSED action_count 94195419013976 step 17479529273785244716 next 2934
    2021-08-09 03:52:38.117806: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3e23440000 of size 510394368 by op gradient_tape/unet_depth4/two_conv_decoder1/conv1d_520/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-
    LayoutOptimizer action_count 94195480112110 step 13755587135625806913 next 3033
    2021-08-09 03:52:38.117832: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3e41b00000 of size 340262912 by op unet_depth4/two_conv_decoder1/conv1d_520/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 9419
    5480112131 step 13755587135625806913 next 2927
    2021-08-09 03:52:38.117861: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3e55f80000 of size 340262912 by op gradient_tape/unet_depth4/two_conv_decoder1/conv1d_521/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-
    LayoutOptimizer action_count 94195480112149 step 13755587135625806913 next 3022
    2021-08-09 03:52:38.117889: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3e6a400000 of size 340262912 by op unet_depth4/two_conv_decoder1/batch_normalization_598/batchnorm/mul_1 action_count 94195480112185 step 13
    755587135625806913 next 2998
    2021-08-09 03:52:38.117922: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3e7e880000 of size 340262912 by op unet_depth4/two_conv_decoder1/conv1d_521/conv1d-0-0-TransposeNCHWToNHWC-LayoutOptimizer action_count 9419
    5480112169 step 13755587135625806913 next 3029
    2021-08-09 03:52:38.117951: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3e92d00000 of size 340262912 by op gradient_tape/unet_depth4/conv_transpose_decoder0/conv1d_transpose_107/conv1d_transpose/Conv2DBackpropFil
    ter-2-TransposeNHWCToNCHW-LayoutOptimizer action_count 94195480112187 step 13755587135625806913 next 2938
    2021-08-09 03:52:38.117978: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ea7180000 of size 340262912 by op unet_depth4/conv_transpose_decoder0/conv1d_transpose_107/conv1d_transpose-0-0-TransposeNCHWToNHWC-LayoutO
    ptimizer action_count 94195480112195 step 13755587135625806913 next 2945
    2021-08-09 03:52:38.118005: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ebb600000 of size 340262912 by op unet_depth4/conv_transpose_decoder0/batch_normalization_599/batchnorm/mul_1 action_count 94195480112211 s
    tep 13755587135625806913 next 3021
    2021-08-09 03:52:38.118031: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] InUse at 2b3ecfa80000 of size 510394368 by op unet_depth4/decoder0/concat action_count 94195480112213 step 13755587135625806913 next 3080
    2021-08-09 03:52:38.118055: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Free  at 2b3eee140000 of size 354680832 by op UNUSED action_count 94195419013931 step 17479529273785244716 next 18446744073709551615
    2021-08-09 03:52:38.118090: I tensorflow/core/common_runtime/bfc_allocator.cc:1051]      Summary of in-use Chunks by size:
    2021-08-09 03:52:38.118121: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1712 Chunks of size 256 totalling 428.0KiB
    2021-08-09 03:52:38.118149: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 140 Chunks of size 512 totalling 70.0KiB
    2021-08-09 03:52:38.118185: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 66 Chunks of size 768 totalling 49.5KiB
    2021-08-09 03:52:38.118212: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 167 Chunks of size 1024 totalling 167.0KiB
    2021-08-09 03:52:38.118237: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 64 Chunks of size 1280 totalling 80.0KiB
    2021-08-09 03:52:38.118262: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 16 Chunks of size 1536 totalling 24.0KiB
    2021-08-09 03:52:38.118290: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 95 Chunks of size 1792 totalling 166.2KiB
    2021-08-09 03:52:38.118326: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 317 Chunks of size 2048 totalling 634.0KiB
    2021-08-09 03:52:38.118354: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 11 Chunks of size 2304 totalling 24.8KiB
    2021-08-09 03:52:38.118377: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 5 Chunks of size 2560 totalling 12.5KiB
    2021-08-09 03:52:38.118402: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 8 Chunks of size 2816 totalling 22.0KiB
    2021-08-09 03:52:38.118437: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 5 Chunks of size 3072 totalling 15.0KiB
    2021-08-09 03:52:38.118463: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 6 Chunks of size 3328 totalling 19.5KiB
    2021-08-09 03:52:38.118488: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 28 Chunks of size 3584 totalling 98.0KiB
    2021-08-09 03:52:38.118511: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 5 Chunks of size 3840 totalling 18.8KiB
    2021-08-09 03:52:38.118554: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 54 Chunks of size 4096 totalling 216.0KiB
    2021-08-09 03:52:38.118581: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 4352 totalling 4.2KiB
    2021-08-09 03:52:38.118605: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 4608 totalling 18.0KiB
    2021-08-09 03:52:38.118639: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 4864 totalling 9.5KiB
    2021-08-09 03:52:38.118666: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 5632 totalling 5.5KiB
    2021-08-09 03:52:38.118693: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 6656 totalling 13.0KiB
    2021-08-09 03:52:38.118716: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 13312 totalling 39.0KiB
    2021-08-09 03:52:38.118752: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 26368 totalling 25.8KiB
    2021-08-09 03:52:38.118782: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 29696 totalling 29.0KiB
    2021-08-09 03:52:38.118812: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 31488 totalling 30.8KiB
    2021-08-09 03:52:38.118849: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 52480 totalling 205.0KiB
    2021-08-09 03:52:38.118878: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 69888 totalling 136.5KiB
    2021-08-09 03:52:38.118904: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 77056 totalling 75.2KiB
    2021-08-09 03:52:38.118943: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 78592 totalling 230.2KiB
    2021-08-09 03:52:38.118971: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 15 Chunks of size 81920 totalling 1.17MiB
    2021-08-09 03:52:38.118999: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 96512 totalling 94.2KiB
    2021-08-09 03:52:38.119035: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 100352 totalling 98.0KiB
    2021-08-09 03:52:38.119063: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 104704 totalling 306.8KiB
    2021-08-09 03:52:38.119091: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 137472 totalling 268.5KiB
    2021-08-09 03:52:38.119127: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 139264 totalling 136.0KiB
    2021-08-09 03:52:38.119155: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 167168 totalling 489.8KiB
    2021-08-09 03:52:38.119183: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 171008 totalling 167.0KiB
    2021-08-09 03:52:38.119219: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 190208 totalling 185.8KiB
    2021-08-09 03:52:38.119247: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 205568 totalling 200.8KiB
    2021-08-09 03:52:38.119276: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 6 Chunks of size 209152 totalling 1.20MiB
    2021-08-09 03:52:38.119308: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 270336 totalling 264.0KiB
    2021-08-09 03:52:38.119334: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 274944 totalling 1.05MiB
    2021-08-09 03:52:38.119360: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 278784 totalling 816.8KiB
    2021-08-09 03:52:38.119396: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 313856 totalling 613.0KiB
    2021-08-09 03:52:38.119424: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 334336 totalling 979.5KiB
    2021-08-09 03:52:38.119450: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 360448 totalling 704.0KiB
    2021-08-09 03:52:38.119487: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 397824 totalling 388.5KiB
    2021-08-09 03:52:38.119514: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 418304 totalling 1.20MiB
    2021-08-09 03:52:38.119557: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 32 Chunks of size 425984 totalling 13.00MiB
    2021-08-09 03:52:38.119582: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 9 Chunks of size 549632 totalling 4.72MiB
    2021-08-09 03:52:38.119607: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 549888 totalling 537.0KiB
    2021-08-09 03:52:38.119642: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 557568 totalling 544.5KiB
    2021-08-09 03:52:38.119667: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 7 Chunks of size 668416 totalling 4.46MiB
    2021-08-09 03:52:38.119702: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 824320 totalling 805.0KiB
    2021-08-09 03:52:38.119728: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 824576 totalling 3.15MiB
    2021-08-09 03:52:38.119751: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 836352 totalling 3.19MiB
    2021-08-09 03:52:38.119786: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 891136 totalling 1.70MiB
    2021-08-09 03:52:38.119815: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 994560 totalling 971.2KiB
    2021-08-09 03:52:38.119839: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 995328 totalling 972.0KiB
    2021-08-09 03:52:38.119872: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 1002752 totalling 1.91MiB
    2021-08-09 03:52:38.119898: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 1054720 totalling 1.01MiB
    2021-08-09 03:52:38.119923: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 1055488 totalling 1.01MiB
    2021-08-09 03:52:38.119959: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 1081344 totalling 2.06MiB
    2021-08-09 03:52:38.119985: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 5 Chunks of size 1099264 totalling 5.24MiB
    2021-08-09 03:52:38.120012: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 1206016 totalling 1.15MiB
    2021-08-09 03:52:38.120035: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 1254656 totalling 3.59MiB
    2021-08-09 03:52:38.120072: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 1336832 totalling 3.82MiB
    2021-08-09 03:52:38.120100: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 13 Chunks of size 1441792 totalling 17.88MiB
    2021-08-09 03:52:38.120127: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 1532928 totalling 1.46MiB
    2021-08-09 03:52:38.120163: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 1622016 totalling 3.09MiB
    2021-08-09 03:52:38.120192: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 1933312 totalling 3.69MiB
    2021-08-09 03:52:38.120218: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 1964800 totalling 1.87MiB
    2021-08-09 03:52:38.120255: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 7 Chunks of size 2097152 totalling 14.00MiB
    2021-08-09 03:52:38.120283: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 9 Chunks of size 2198272 totalling 18.87MiB
    2021-08-09 03:52:38.120310: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 6 Chunks of size 2673408 totalling 15.30MiB
    2021-08-09 03:52:38.120345: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 2723840 totalling 2.60MiB
    2021-08-09 03:52:38.120373: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 2899968 totalling 8.30MiB
    2021-08-09 03:52:38.120400: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 6 Chunks of size 2930944 totalling 16.77MiB
    2021-08-09 03:52:38.120437: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 22 Chunks of size 3145728 totalling 66.00MiB
    2021-08-09 03:52:38.120465: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 6 Chunks of size 3297536 totalling 18.87MiB
    2021-08-09 03:52:38.120491: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 3714816 totalling 3.54MiB
    2021-08-09 03:52:38.120534: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 4010240 totalling 11.47MiB
    2021-08-09 03:52:38.120560: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 4046080 totalling 3.86MiB
    2021-08-09 03:52:38.120586: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 5 Chunks of size 4194304 totalling 20.00MiB
    2021-08-09 03:52:38.120621: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 4396544 totalling 8.39MiB
    2021-08-09 03:52:38.120647: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 4673536 totalling 4.46MiB
    2021-08-09 03:52:38.120672: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 4767744 totalling 13.64MiB
    2021-08-09 03:52:38.120696: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 4894464 totalling 4.67MiB
    2021-08-09 03:52:38.120728: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 5318400 totalling 5.07MiB
    2021-08-09 03:52:38.120755: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 5426944 totalling 5.17MiB
    2021-08-09 03:52:38.120780: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 5647360 totalling 5.39MiB
    2021-08-09 03:52:38.120805: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 5 Chunks of size 5775360 totalling 27.54MiB
    2021-08-09 03:52:38.120841: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 3 Chunks of size 6045696 totalling 17.30MiB
    2021-08-09 03:52:38.120866: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 6225920 totalling 5.94MiB
    2021-08-09 03:52:38.120891: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 12 Chunks of size 6291456 totalling 72.00MiB
    2021-08-09 03:52:38.120925: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 6381568 totalling 6.09MiB
    2021-08-09 03:52:38.120953: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 6754048 totalling 6.44MiB
    2021-08-09 03:52:38.120978: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 6 Chunks of size 7012352 totalling 40.12MiB
    2021-08-09 03:52:38.121003: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 7193600 totalling 6.86MiB
    2021-08-09 03:52:38.121036: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 7728640 totalling 7.37MiB
    2021-08-09 03:52:38.121062: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 8192000 totalling 15.62MiB
    2021-08-09 03:52:38.121088: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 8792832 totalling 33.54MiB
    2021-08-09 03:52:38.121111: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 9883392 totalling 9.42MiB
    2021-08-09 03:52:38.121146: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 10772736 totalling 10.27MiB
    2021-08-09 03:52:38.121174: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 4 Chunks of size 12582912 totalling 48.00MiB
    2021-08-09 03:52:38.121199: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 13459968 totalling 12.84MiB
    2021-08-09 03:52:38.121232: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 5 Chunks of size 14024704 totalling 66.88MiB
    2021-08-09 03:52:38.121259: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 14162944 totalling 13.51MiB
    2021-08-09 03:52:38.121288: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 18874368 totalling 18.00MiB
    2021-08-09 03:52:38.121323: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 24661504 totalling 23.52MiB
    2021-08-09 03:52:38.121350: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 36044800 totalling 68.75MiB
    2021-08-09 03:52:38.121373: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 46137344 totalling 44.00MiB
    2021-08-09 03:52:38.121406: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 78643200 totalling 75.00MiB
    2021-08-09 03:52:38.121434: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 85035008 totalling 81.10MiB
    2021-08-09 03:52:38.121473: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 5 Chunks of size 85065728 totalling 405.62MiB
    2021-08-09 03:52:38.121506: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 20 Chunks of size 92274688 totalling 1.72GiB
    2021-08-09 03:52:38.121535: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 143653632 totalling 137.00MiB
    2021-08-09 03:52:38.121560: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 157947392 totalling 150.63MiB
    2021-08-09 03:52:38.121594: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 170129920 totalling 162.25MiB
    2021-08-09 03:52:38.121619: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 16 Chunks of size 170131456 totalling 2.54GiB
    2021-08-09 03:52:38.121646: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 8 Chunks of size 184549376 totalling 1.38GiB
    2021-08-09 03:52:38.121671: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 245764096 totalling 234.38MiB
    2021-08-09 03:52:38.121706: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 262275072 totalling 250.12MiB
    2021-08-09 03:52:38.121732: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 276037120 totalling 263.25MiB
    2021-08-09 03:52:38.121757: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 314572800 totalling 600.00MiB
    2021-08-09 03:52:38.121789: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 11 Chunks of size 340262912 totalling 3.49GiB
    2021-08-09 03:52:38.121817: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 1 Chunks of size 354680832 totalling 338.25MiB
    2021-08-09 03:52:38.121844: I tensorflow/core/common_runtime/bfc_allocator.cc:1054] 2 Chunks of size 510394368 totalling 973.50MiB
    2021-08-09 03:52:38.121879: I tensorflow/core/common_runtime/bfc_allocator.cc:1058] Sum Total of in-use chunks: 13.57GiB
    2021-08-09 03:52:38.121906: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] total_region_allocated_bytes_: 15354822656 memory_limit_: 15354822656 available bytes: 0 curr_region_allocation_bytes_: 17179869184
    2021-08-09 03:52:38.121937: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Stats:
    Limit:                     15354822656
    InUse:                     14574896128
    MaxInUse:                  14574896128
    NumAllocs:                   225920650
    MaxAllocSize:               3306898944
    Reserved:                            0
    PeakReserved:                        0
    LargestFreeBlock:                    0
    
    2021-08-09 03:52:38.122399: W tensorflow/core/common_runtime/bfc_allocator.cc:467] **************************************************************************************************__
    2021-08-09 03:52:38.122475: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at transpose_op.cc:184 : Resource exhausted: OOM when allocating tensor with shape[22,354,1,16384] and type float on /job:localhost/replica:0
    /task:0/device:GPU:0 by allocator GPU_0_bfc
    Traceback (most recent call last):
      File "/beegfs/ye53nis/drmed-git/src/fluotracify/training/search_hparams.py", line 446, in <module>
        hparams_run()
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
        return self.main(*args, **kwargs)
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/click/core.py", line 1062, in main
        rv = self.invoke(ctx)
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/click/core.py", line 763, in invoke
        return __callback(*args, **kwargs)
      File "/beegfs/ye53nis/drmed-git/src/fluotracify/training/search_hparams.py", line 405, in hparams_run
        best_auc_val = run_one(
      File "/beegfs/ye53nis/drmed-git/src/fluotracify/training/search_hparams.py", line 306, in run_one
        result = model.fit(
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/mlflow/utils/autologging_utils/safety.py", line 490, in safe_patch_function
        patch_function.call(call_original, *args, **kwargs)
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/mlflow/utils/autologging_utils/safety.py", line 156, in call
        return cls().__call__(original, *args, **kwargs)
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/mlflow/utils/autologging_utils/safety.py", line 167, in __call__
        raise e
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/mlflow/utils/autologging_utils/safety.py", line 160, in __call__
        return self._patch_implementation(original, *args, **kwargs)
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/mlflow/utils/autologging_utils/safety.py", line 218, in _patch_implementation
        result = super(PatchWithManagedRun, self)._patch_implementation(
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/mlflow/tensorflow.py", line 1097, in _patch_implementation
        history = original(inst, *args, **kwargs)
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/mlflow/utils/autologging_utils/safety.py", line 448, in call_original
        original_result = original(*og_args, **og_kwargs)
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/tensorflow/python/keras/engine/training.py", line 1183, in fit
        tmp_logs = self.train_function(iterator)
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
        result = self._call(*args, **kwds)
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 950, in _call
        return self._stateless_fn(*args, **kwds)
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3023, in __call__
        return graph_function._call_flat(
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 1960, in _call_flat
        return self._build_call_outputs(self._inference_function.call(
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 591, in call
        outputs = execute.execute(
      File "/home/ye53nis/.conda/envs/mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
        tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
    tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
      (0) Resource exhausted:  OOM when allocating tensor with shape[22,354,1,16384] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
             [[{{node gradient_tape/unet_depth4/two_conv_decoder0/conv1d_522/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer}}]]
    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
    
             [[assert_less_equal_4/Assert/AssertGuard/pivot_f/_125/_159]]
    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
    
      (1) Resource exhausted:  OOM when allocating tensor with shape[22,354,1,16384] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
             [[{{node gradient_tape/unet_depth4/two_conv_decoder0/conv1d_522/conv1d/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer}}]]
    Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
    
    0 successful operations.
    0 derived errors ignored. [Op:__inference_train_function_65983138]
    
    Function call stack:
    train_function -> train_function
    
    2021/08/09 03:52:57 ERROR mlflow.cli: === Run (ID 'b6cbd5623eb44c6c8158c97b7a40c651') failed ===
    (tf) [ye53nis@node130 drmed-git]$
    

2.4.4 Analyze run 1 and 2

  1. start
    %cd /beegfs/ye53nis/drmed-git
    
    /beegfs/ye53nis/drmed-git
    
    import mlflow
    import sys
    
    import matplotlib.pyplot as plt
    import numpy as np
    import pandas as pd
    
    sys.path.append('src/')
    from fluotracify.training import build_model as bm, preprocess_data as ppd
    from fluotracify.applications import correlate, plots, correction
    from fluotracify.imports import ptu_utils as ptu
    
  2. Out of convenience, I used Mlflowui to compare the two runs with all parameters and metrics. Since a nice exporting option was missing, I copied the comparison into libreoffice calc by hand and saved it as a csv. Now lets load it and print the validation AUC as an example.
    run1_2 = pd.read_csv('data/exp-210807-hparams/run1-2_comparison.csv', index_col=0)
    run1_2_valauc = run1_2.loc['val_auc'].astype(float)
    run1_2_valauc
    
    2761cb4ad23244beafac36189b805c3a    0.975
    d92021ec45f5415283a8e5643c7e3449    0.975
    e6b70d64227f42bcb647289d2af2afb4    0.944
    876fa57de00643b1902f3a0be74e0682    0.940
    29905abbb90344798ee59f1f40775880    0.929
                                        ...
    158bfabc0fcc448dada729967f924574    0.969
    3423b9fa10e44eb7b2cffd78596a193f    0.897
    985ec48a8e1b4101abb8a153cee69b57    0.891
    5c3e5090c67f4340bf207f949e925758    0.601
    ff0bc51f1cd3413bbdebb97b0e127e75    0.889
    Name: val_auc, Length: 75, dtype: float64
    
  3. I ran each hparams run twice for each random parameter sample. As a minimal statistical move, lets take the average of the metrics which have 2 identical parameter runs. Unfortunately, one run is only single, since the program crashed. We have to remove it and later add it again.
    singles_ls = ['5441e71efe0f4dae868648e7cc795c65']
    run1_2_singles = run1_2.loc[:, singles_ls]
    run1_2_singles.iloc[35:, :] = run1_2_singles.iloc[35:, :].astype(np.float64)
    run1_2_singles
    
    5441e71efe0f4dae868648e7cc795c65
    Run ID:
    Run Name: NaN
    Start Time: 2021-08-08 13:46:29
    batch_size None
    class_weight None
    epochs 20
    ... ...
    val_tp0.1 756740.0
    val_tp0.3 686440.0
    val_tp0.5 622523.0
    val_tp0.7 505054.0
    val_tp0.9 241926.0

    104 rows × 1 columns

    run1_2 = run1_2.drop(columns=singles_ls)
    
    assert len(run1_2.iloc[35:, :].columns) % 2 == 0
    
    run1_2_doubleparams = pd.DataFrame()
    run1_2_doublemetrics = pd.DataFrame()
    double_cols = []
    for left, right in zip(run1_2.iloc[:, ::2].items(), run1_2.iloc[:, 1::2].items()):
        double_cols.append((left[0], right[0]))
        current_metrics = left[1].iloc[35:].combine(other=right[1].iloc[35:],
                                                    func=(lambda x1, x2: (float(x1) + float(x2)) / 2))
        current_params = left[1].iloc[:35].combine(other=right[1].iloc[:35],
                                                   func=(lambda x1, x2: set((x1, x2)) if x1 != x2 else x1))
        run1_2_doubleparams = pd.concat([run1_2_doubleparams, current_params], axis=1)
        run1_2_doublemetrics = pd.concat([run1_2_doublemetrics, current_metrics], axis=1)
    
    run1_2_doublemetrics = pd.DataFrame(data=run1_2_doublemetrics.to_numpy(),
                                        index=run1_2.iloc[35:, :].index,
                                        columns=double_cols)
    
    run1_2_doubleparams = pd.DataFrame(data=run1_2_doubleparams.to_numpy(),
                                       index=run1_2.iloc[:35, :].index,
                                       columns=double_cols)
    
    run1_2_combimetrics = pd.concat([run1_2_doublemetrics, run1_2_singles.iloc[35:, :]], axis=1)
    run1_2_combiparams = pd.concat([run1_2_doubleparams, run1_2_singles.iloc[:35, :]], axis=1)
    run1_2_combiparams
    
    (2761cb4ad23244beafac36189b805c3a, d92021ec45f5415283a8e5643c7e3449) (e6b70d64227f42bcb647289d2af2afb4, 876fa57de00643b1902f3a0be74e0682) (29905abbb90344798ee59f1f40775880, 5a89360285384c728e9e06421ab97c8d) (ab0ff6fff89c4db6983aa98c7eee9663, b9d383f0e9ef4755a801376130594f2a) (481bacb91dcb4d63b16d0a8fe5d198ec, 2ef6df0c95b34b338c4b930f41695ed0) (d36564aa214c41bc966fc7b890df31a8, ba34b505ba324f208d428bd7797233e7) (0a484d3116ab436d8b4cf4a938d470c0, 5a2dfcd776014637bdaef2da3ab1d132) (b72a292b214c462480cbe2e66e811c78, 50c7e9bba0534a27801143bda2e88d35) (9051e32b87d84f3485b980067addec30, 61ff87bdb89b4e2ba64f8dacc774992d) (93b168c0ff7942c8a908a94129daf973, f243b3b742de4dbcb7ccfbd4244706f8) ... (3cbd945b62ec4634839372e403f6f377, 458b36a70db843719d202a8eda448f17) (2648b86e7ef54a0e9f7b57340703150e, 0aae5a802f5342cebed9e51839d48b41) (7719d9fbef8f4c22b710caf91cd527d2, 2b7aab186fad48158d53b78610550d73) (5bbaee372809482ea6d2d985b554cc4b, b8200ec9fb884c12b390e1a5b20299bc) (37ab5992ecdf4760a6292bc64e61e309, 91e4770c10ac4cceb405bfdcebf7c966) (90a949129948495fb7f3b955e8258461, ce09d20be06745fb84c9ff177c8a9cef) (d320b958bde94b1e998d210fd2e53efa, 158bfabc0fcc448dada729967f924574) (3423b9fa10e44eb7b2cffd78596a193f, 985ec48a8e1b4101abb8a153cee69b57) (5c3e5090c67f4340bf207f949e925758, ff0bc51f1cd3413bbdebb97b0e127e75) 5441e71efe0f4dae868648e7cc795c65
    Run ID:
    Run Name: {nan} {nan} {nan} {nan} {nan} {nan} {nan} {nan} {nan} {nan} ... {nan} {nan} {nan} {nan} {nan} {nan} {nan} {nan} {nan} NaN
    Start Time: {2021-08-09 03:35:27, 2021-08-09 03:18:38} {2021-08-09 03:00:40, 2021-08-09 02:43:08} {2021-08-09 02:21:19, 2021-08-09 01:59:18} {2021-08-09 01:45:12, 2021-08-09 01:52:15} {2021-08-09 01:01:03, 2021-08-09 00:16:49} {2021-08-09 00:01:24, 2021-08-08 23:46:06} {2021-08-08 23:09:26, 2021-08-08 23:27:50} {2021-08-08 23:00:25, 2021-08-08 22:51:22} {2021-08-08 22:10:33, 2021-08-08 21:29:38} {2021-08-08 21:15:24, 2021-08-08 21:01:49} ... {2021-08-08 04:30:30, 2021-08-08 04:19:31} {2021-08-08 03:58:55, 2021-08-08 03:38:13} {2021-08-08 03:12:28, 2021-08-08 03:25:26} {2021-08-08 03:05:58, 2021-08-08 02:59:20} {2021-08-08 02:04:14, 2021-08-08 01:12:38} {2021-08-08 01:05:46, 2021-08-08 00:58:39} {2021-08-08 00:26:43, 2021-08-08 00:43:13} {2021-08-07 23:57:58, 2021-08-07 23:31:44} {2021-08-07 22:57:43, 2021-08-07 22:23:42} 2021-08-08 13:46:29
    batch_size None None None None None None None None None None ... None None None None None None None None None None
    class_weight None None None None None None None None None None ... None None None None None None None None None None
    epochs 20 20 20 20 20 20 20 20 20 20 ... 20 20 20 20 20 20 20 20 20 20
    hp_batch_size 13 5 11 23 13 13 4 18 26 15 ... 9 4 18 10 4 20 14 5 18 14
    hp_epochs 20 20 20 20 20 20 20 20 20 20 ... 20 20 20 20 20 20 20 20 20 20
    hp_first_filters 107 33 102 58 108 41 64 59 44 23 ... 64 128 32 16 64 32 128 64 32 16
    hp_input_size 8192 4096 16384 8192 8192 16384 4096 4096 16384 16384 ... 4096 4096 16384 4096 8192 4096 4096 16384 16384 4096
    hp_lr_power 5 7 6 6 6 6 2 2 1 7 ... 1 5 5 1 1 5 1 5 1 5
    hp_lr_start 0.00782433731433605 0.0183838908333744 0.0482033274205359 0.00878855796222036 0.00652659278503235 0.0571175071464742 0.047837465592254 0.0330458251501699 0.0136170138242663 0.0305060808685107 ... 0.0100697459464075 0.024311131849965 0.0275283294963226 0.0923388677844686 0.0183950335666331 0.00805377739706802 0.0730305548495589 0.0100791137947559 0.0799422633823613 0.0271446293545328
    hp_n_levels 3 5 1 1 6 3 5 1 7 6 ... 5 3 5 3 9 5 3 3 7 7
    hp_pool_size 8 2 8 2 2 4 4 8 2 4 ... 4 4 4 2 2 4 4 4 2 2
    hp_scaler maxabs robust quant_g minmax minmax l1 standard quant_g standard quant_g ... maxabs maxabs maxabs minmax quant_g l2 robust robust l2 quant_g
    initial_epoch 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
    lr schedule [0.00782433731433605, 0.0060543230624031785, 0... [0.01838389083337442, 0.012838156616261365, 0.... [0.04820332742053588, 0.035433875087977615, 0.... [0.00878855796222036, 0.0064603976883159595, 0... [0.0065265927850323546, 0.004797645429688916, ... [0.057117507146474185, 0.041986616316088646, 0... [0.04783746559225395, 0.043173312697009185, 0.... [0.03304582515016993, 0.029823857198028363, 0.... [0.01361701382426631, 0.012936163133052994, 0.... [0.030506080868510657, 0.021303534028133002, 0... ... [0.010069745946407459, 0.009566258649087086, 0... [0.024311131849965008, 0.018811490394552027, 0... [0.02752832949632261, 0.021300896605473404, 0.... [0.0923388677844686, 0.08772192439524518, 0.08... [0.018395033566633148, 0.01747528188830149, 0.... [0.008053777397068015, 0.0062318594247195965, ... [0.07303055484955895, 0.06937902710708099, 0.0... [0.010079113794755897, 0.007799026121275398, 0... [0.0799422633823613, 0.07594515021324323, 0.07... [0.027144629354532844, 0.021003996750040436, 0...
    max_queue_size 10 10 10 10 10 10 10 10 10 10 ... 10 10 10 10 10 10 10 10 10 10
    num_train_examples 4800 4800 4800 4800 4800 4800 4800 4800 4800 4800 ... 4800 4800 4800 4800 4800 4800 4800 4800 4800 4800
    num_val_examples 1200 1200 1200 1200 1200 1200 1200 1200 1200 1200 ... 1200 1200 1200 1200 1200 1200 1200 1200 1200 1200
    opt_amsgrad False False False False False False False False False False ... False False False False False False False False False False
    opt_beta_1 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 ... 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9
    opt_beta_2 0.999 0.999 0.999 0.999 0.999 0.999 0.999 0.999 0.999 0.999 ... 0.999 0.999 0.999 0.999 0.999 0.999 0.999 0.999 0.999 0.999
    opt_decay 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
    opt_epsilon 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 ... 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001
    opt_learning_rate 0.001 0.001 0.001 0.001 0.001 0.001 0.001 0.001 0.001 0.001 ... 0.001 0.001 0.001 0.001 0.001 0.001 0.001 0.001 0.001 0.001
    opt_name Adam Adam Adam Adam Adam Adam Adam Adam Adam Adam ... Adam Adam Adam Adam Adam Adam Adam Adam Adam Adam
    sample_weight None None None None None None None None None None ... None None None None None None None None None None
    shuffle True True True True True True True True True True ... True True True True True True True True True True
    steps_per_epoch 369 960 436 208 369 369 1200 266 184 320 ... 533 1200 266 480 1200 240 342 960 266 342
    use_multiprocessing False False False False False False False False False False ... False False False False False False False False False False
    validation_batch_size None None None None None None None None None None ... None None None None None None None None None None
    validation_freq 1 1 1 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1 1 1 1
    validation_split 0 0 0 0 0 0 0 0 0 0 ... 0 0 0 0 0 0 0 0 0 0
    validation_steps 92 240 109 52 92 92 300 66 46 80 ... 133 300 66 120 300 60 85 240 66 85
    workers 1 1 1 1 1 1 1 1 1 1 ... 1 1 1 1 1 1 1 1 1 1

    35 rows × 38 columns

  4. Now choose the "most successful" runs. I don't just choose a high AUC, but especially a high Recall as well:
    cond1 = run1_2_combimetrics.loc['val_auc'] > 0.95
    cond2 = run1_2_combimetrics.loc['val_recall0.5'] > 0.85
    cond3 = run1_2_combimetrics.loc['val_precision0.5'] > 0.85
    
    with pd.option_context('display.max_rows', None, 'display.max_columns', None):  # more options can be specified also
        display(run1_2_combiparams.loc[:, cond1 & cond2 & cond3])
        display(run1_2_combimetrics.loc[:, cond1 & cond2 & cond3])
    
    (9051e32b87d84f3485b980067addec30, 61ff87bdb89b4e2ba64f8dacc774992d) (93b168c0ff7942c8a908a94129daf973, f243b3b742de4dbcb7ccfbd4244706f8) (a5b8551144ff46e697a39cd1551e1475, 98cf8cdef9c54b5286e277e75e2ab8c1) (00f2635d9fa2463c9a066722163405be, d0a8e1748b194f3290d471b6b44f19f8) (5604d43c1ece461b8e6eaa0dfb65d6dc, 3612536a77f34f22bc83d1d809140aa6) (7cafab027cdd4fc9bf20a43e989df510, 16dff15d935f45e2a836b1f41b07b4e3) (0e328920e86049928202db95e8cfb7be, bf9d2725eb16462d9a101f0a077ce2b5) (1c954fbc02b747bc813c587ac703c74a, ba49a80c2616407a8f1fe1fd12096fe0) (3cbd945b62ec4634839372e403f6f377, 458b36a70db843719d202a8eda448f17)
    Run ID:
    Run Name: {nan} {nan} {nan} {nan} {nan} {nan} {nan} {nan} {nan}
    Start Time: {2021-08-08 22:10:33, 2021-08-08 21:29:38} {2021-08-08 21:15:24, 2021-08-08 21:01:49} {2021-08-08 19:41:51, 2021-08-08 19:14:27} {2021-08-08 19:07:59, 2021-08-08 19:01:30} {2021-08-08 12:21:05, 2021-08-08 13:03:41} {2021-08-08 10:37:39, 2021-08-08 10:29:07} {2021-08-08 07:47:04, 2021-08-08 08:10:44} {2021-08-08 05:49:55, 2021-08-08 05:42:04} {2021-08-08 04:30:30, 2021-08-08 04:19:31}
    batch_size None None None None None None None None None
    class_weight None None None None None None None None None
    epochs 20 20 20 20 20 20 20 20 20
    hp_batch_size 26 15 20 28 20 10 14 17 9
    hp_epochs 20 20 20 20 20 20 20 20 20
    hp_first_filters 44 23 78 6 128 16 16 16 64
    hp_input_size 16384 16384 16384 16384 16384 8192 16384 16384 4096
    hp_lr_power 1 7 4 1 1 1 5 5 1
    hp_lr_start 0.0136170138242663 0.0305060808685107 0.0584071108418767 0.0553313915596308 0.043549707353273 0.0627676336651573 0.0192390310290551 0.0101590069352232 0.0100697459464075
    hp_n_levels 7 6 4 5 3 5 9 3 5
    hp_pool_size 2 4 4 4 4 4 2 4 4
    hp_scaler standard quant_g standard minmax standard robust robust l2 maxabs
    initial_epoch 0 0 0 0 0 0 0 0 0
    lr schedule [0.01361701382426631, 0.012936163133052994, 0.... [0.030506080868510657, 0.021303534028133002, 0... [0.05840711084187669, 0.04757295682515132, 0.0... [0.05533139155963077, 0.05256482198164923, 0.0... [0.04354970735327304, 0.041372221985609386, 0.... [0.06276763366515732, 0.05962925198189945, 0.0... [0.019239031029055137, 0.014886795466253868, 0... [0.010159006935223234, 0.007860845910406034, 0... [0.010069745946407459, 0.009566258649087086, 0...
    max_queue_size 10 10 10 10 10 10 10 10 10
    num_train_examples 4800 4800 4800 4800 4800 4800 4800 4800 4800
    num_val_examples 1200 1200 1200 1200 1200 1200 1200 1200 1200
    opt_amsgrad False False False False False False False False False
    opt_beta_1 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9
    opt_beta_2 0.999 0.999 0.999 0.999 0.999 0.999 0.999 0.999 0.999
    opt_decay 0 0 0 0 0 0 0 0 0
    opt_epsilon 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001 0.0000001
    opt_learning_rate 0.001 0.001 0.001 0.001 0.001 0.001 0.001 0.001 0.001
    opt_name Adam Adam Adam Adam Adam Adam Adam Adam Adam
    sample_weight None None None None None None None None None
    shuffle True True True True True True True True True
    steps_per_epoch 184 320 240 171 240 480 342 282 533
    use_multiprocessing False False False False False False False False False
    validation_batch_size None None None None None None None None None
    validation_freq 1 1 1 1 1 1 1 1 1
    validation_split 0 0 0 0 0 0 0 0 0
    validation_steps 46 80 60 42 60 120 85 70 133
    workers 1 1 1 1 1 1 1 1 1
    (9051e32b87d84f3485b980067addec30, 61ff87bdb89b4e2ba64f8dacc774992d) (93b168c0ff7942c8a908a94129daf973, f243b3b742de4dbcb7ccfbd4244706f8) (a5b8551144ff46e697a39cd1551e1475, 98cf8cdef9c54b5286e277e75e2ab8c1) (00f2635d9fa2463c9a066722163405be, d0a8e1748b194f3290d471b6b44f19f8) (5604d43c1ece461b8e6eaa0dfb65d6dc, 3612536a77f34f22bc83d1d809140aa6) (7cafab027cdd4fc9bf20a43e989df510, 16dff15d935f45e2a836b1f41b07b4e3) (0e328920e86049928202db95e8cfb7be, bf9d2725eb16462d9a101f0a077ce2b5) (1c954fbc02b747bc813c587ac703c74a, ba49a80c2616407a8f1fe1fd12096fe0) (3cbd945b62ec4634839372e403f6f377, 458b36a70db843719d202a8eda448f17)
    Run ID:
    accuracy 9.710000e-01 9.600000e-01 9.725000e-01 9.735000e-01 9.665000e-01 9.705000e-01 9.640000e-01 9.615000e-01 9.705000e-01
    auc 9.810000e-01 9.730000e-01 9.820000e-01 9.840000e-01 9.720000e-01 9.810000e-01 9.730000e-01 9.625000e-01 9.800000e-01
    f1 2.640000e-01 2.640000e-01 2.640000e-01 2.650000e-01 2.640000e-01 2.620000e-01 2.650000e-01 2.635000e-01 2.630000e-01
    fn0.1 3.682675e+05 4.621365e+05 3.568190e+05 3.122355e+05 5.600420e+05 1.845900e+05 5.070725e+05 7.945210e+05 1.017630e+05
    fn0.3 7.227145e+05 9.977800e+05 6.926625e+05 6.512465e+05 9.973125e+05 3.611175e+05 1.124122e+06 1.259410e+06 1.843830e+05
    fn0.5 1.136268e+06 1.521807e+06 1.071660e+06 1.004638e+06 1.420861e+06 5.911320e+05 1.642768e+06 1.763400e+06 2.860370e+05
    fn0.7 1.738792e+06 2.319607e+06 1.661646e+06 1.556141e+06 2.051528e+06 8.906325e+05 2.305407e+06 2.409087e+06 4.321115e+05
    fn0.9 2.890862e+06 4.209348e+06 2.778182e+06 2.663333e+06 3.247781e+06 1.448880e+06 3.677228e+06 3.719575e+06 7.203375e+05
    fp0.1 3.958967e+06 5.572852e+06 3.626632e+06 3.504270e+06 4.532016e+06 1.989762e+06 5.707579e+06 4.813038e+06 9.582675e+05
    fp0.3 1.991625e+06 2.699346e+06 1.887159e+06 1.789370e+06 2.098466e+06 1.039760e+06 2.281552e+06 2.328430e+06 5.024380e+05
    fp0.5 1.120762e+06 1.617764e+06 1.093034e+06 1.072950e+06 1.197675e+06 5.652505e+05 1.199354e+06 1.254754e+06 2.843205e+05
    fp0.7 5.437065e+05 8.631090e+05 5.284415e+05 5.297935e+05 5.737150e+05 2.738040e+05 5.788060e+05 6.135885e+05 1.421470e+05
    fp0.9 1.503025e+05 2.542905e+05 1.457025e+05 1.390835e+05 1.532515e+05 7.276600e+04 1.507885e+05 1.525270e+05 3.853750e+04
    loss 1.950000e-01 2.620000e-01 1.850000e-01 1.710000e-01 2.485000e-01 1.955000e-01 2.645000e-01 3.070000e-01 2.005000e-01
    lr 6.809000e-04 2.383000e-11 3.650000e-07 3.000000e-03 2.000000e-03 3.000000e-03 6.012000e-09 3.175000e-09 5.035000e-04
    precision0.1 7.455000e-01 6.740000e-01 7.620000e-01 7.690000e-01 7.160000e-01 7.430000e-01 6.675000e-01 6.980000e-01 7.500000e-01
    precision0.3 8.490000e-01 8.030000e-01 8.565000e-01 8.640000e-01 8.395000e-01 8.425000e-01 8.260000e-01 8.205000e-01 8.470000e-01
    precision0.5 9.060000e-01 8.660000e-01 9.090000e-01 9.110000e-01 8.980000e-01 9.045000e-01 8.960000e-01 8.900000e-01 9.040000e-01
    precision0.7 9.495000e-01 9.180000e-01 9.510000e-01 9.515000e-01 9.455000e-01 9.485000e-01 9.435000e-01 9.395000e-01 9.470000e-01
    precision0.9 9.835000e-01 9.680000e-01 9.840000e-01 9.850000e-01 9.825000e-01 9.840000e-01 9.825000e-01 9.815000e-01 9.830000e-01
    recall0.1 9.690000e-01 9.615000e-01 9.700000e-01 9.740000e-01 9.530000e-01 9.690000e-01 9.575000e-01 9.335000e-01 9.660000e-01
    recall0.3 9.395000e-01 9.165000e-01 9.420000e-01 9.455000e-01 9.170000e-01 9.390000e-01 9.060000e-01 8.940000e-01 9.380000e-01
    recall0.5 9.050000e-01 8.730000e-01 9.105000e-01 9.160000e-01 8.815000e-01 9.005000e-01 8.625000e-01 8.520000e-01 9.040000e-01
    recall0.7 8.545000e-01 8.065000e-01 8.610000e-01 8.705000e-01 8.285000e-01 8.500000e-01 8.070000e-01 7.980000e-01 8.545000e-01
    recall0.9 7.580000e-01 6.485000e-01 7.675000e-01 7.775000e-01 7.285000e-01 7.560000e-01 6.925000e-01 6.880000e-01 7.575000e-01
    tn0.1 6.248249e+07 6.110056e+07 6.304679e+07 6.295906e+07 6.214143e+07 3.139506e+07 6.077990e+07 6.181710e+07 1.572065e+07
    tn0.3 6.444982e+07 6.397410e+07 6.478628e+07 6.467399e+07 6.457496e+07 3.234507e+07 6.420593e+07 6.430173e+07 1.617648e+07
    tn0.5 6.532068e+07 6.505565e+07 6.558039e+07 6.539041e+07 6.547576e+07 3.281958e+07 6.528812e+07 6.537538e+07 1.639460e+07
    tn0.7 6.589775e+07 6.581032e+07 6.614497e+07 6.593356e+07 6.609972e+07 3.311101e+07 6.590870e+07 6.601654e+07 1.653677e+07
    tn0.9 6.629115e+07 6.641916e+07 6.652773e+07 6.632426e+07 6.652018e+07 3.331206e+07 6.633670e+07 6.647764e+07 1.664038e+07
    tp0.1 1.157134e+07 1.150763e+07 1.161295e+07 1.167101e+07 1.140972e+07 5.752186e+06 1.145204e+07 1.112024e+07 2.867828e+06
    tp0.3 1.121689e+07 1.097199e+07 1.127710e+07 1.133200e+07 1.097245e+07 5.575658e+06 1.083499e+07 1.065535e+07 2.785208e+06
    tp0.5 1.080334e+07 1.044796e+07 1.089811e+07 1.097861e+07 1.054891e+07 5.345644e+06 1.031634e+07 1.015136e+07 2.683554e+06
    tp0.7 1.020082e+07 9.650160e+06 1.030812e+07 1.042710e+07 9.918238e+06 5.046144e+06 9.653704e+06 9.505670e+06 2.537480e+06
    tp0.9 9.048746e+06 7.760420e+06 9.191586e+06 9.319910e+06 8.721986e+06 4.487896e+06 8.281883e+06 8.195182e+06 2.249254e+06
    val_accuracy 9.705000e-01 9.580000e-01 9.715000e-01 9.730000e-01 9.655000e-01 9.650000e-01 9.620000e-01 9.540000e-01 9.655000e-01
    val_auc 9.810000e-01 9.760000e-01 9.840000e-01 9.870000e-01 9.745000e-01 9.780000e-01 9.760000e-01 9.620000e-01 9.720000e-01
    val_f1 2.765000e-01 2.770000e-01 2.770000e-01 2.765000e-01 2.770000e-01 2.770000e-01 2.775000e-01 2.775000e-01 2.810000e-01
    val_fn0.1 9.953900e+04 1.018855e+05 8.291750e+04 6.491900e+04 1.300915e+05 5.183400e+04 1.083685e+05 1.989905e+05 4.027450e+04
    val_fn0.3 2.065230e+05 2.225190e+05 1.673150e+05 1.418270e+05 2.487260e+05 1.058955e+05 2.645855e+05 3.249370e+05 6.900850e+04
    val_fn0.5 3.215935e+05 3.389065e+05 2.659300e+05 2.205875e+05 3.629365e+05 1.735790e+05 3.913830e+05 4.524360e+05 1.028930e+05
    val_fn0.7 4.900970e+05 5.129575e+05 4.164020e+05 3.444560e+05 5.346575e+05 2.454170e+05 5.597010e+05 6.320105e+05 1.485405e+05
    val_fn0.9 8.184030e+05 9.663815e+05 7.104005e+05 6.000115e+05 8.757840e+05 3.997180e+05 9.522080e+05 9.742990e+05 2.333430e+05
    val_fp0.1 9.453770e+05 1.601886e+06 9.806695e+05 1.017959e+06 1.234908e+06 6.305910e+05 1.691592e+06 1.263437e+06 2.185595e+05
    val_fp0.3 4.490195e+05 8.010390e+05 5.028105e+05 4.986330e+05 5.470645e+05 3.102420e+05 6.715845e+05 7.084290e+05 1.185620e+05
    val_fp0.5 2.523625e+05 4.911005e+05 2.909235e+05 2.976760e+05 3.176580e+05 1.722175e+05 3.474940e+05 4.435775e+05 6.640450e+04
    val_fp0.7 1.222830e+05 2.638220e+05 1.435500e+05 1.506035e+05 1.579470e+05 9.873550e+04 1.668495e+05 2.509655e+05 3.329950e+04
    val_fp0.9 3.509800e+04 7.686200e+04 4.052800e+04 4.137150e+04 4.698050e+04 3.210800e+04 4.534800e+04 9.132200e+04 9.003000e+03
    val_loss 1.985000e-01 2.665000e-01 1.855000e-01 1.735000e-01 2.515000e-01 2.360000e-01 2.725000e-01 3.290000e-01 2.440000e-01
    val_precision0.1 7.635000e-01 6.570000e-01 7.585000e-01 7.495000e-01 7.105000e-01 7.095000e-01 6.420000e-01 6.995000e-01 7.770000e-01
    val_precision0.3 8.675000e-01 7.860000e-01 8.565000e-01 8.560000e-01 8.425000e-01 8.270000e-01 8.110000e-01 7.990000e-01 8.610000e-01
    val_precision0.5 9.180000e-01 8.520000e-01 9.090000e-01 9.065000e-01 8.985000e-01 8.910000e-01 8.880000e-01 8.585000e-01 9.135000e-01
    val_precision0.7 9.560000e-01 9.095000e-01 9.505000e-01 9.480000e-01 9.435000e-01 9.315000e-01 9.395000e-01 9.090000e-01 9.515000e-01
    val_precision0.9 9.850000e-01 9.660000e-01 9.840000e-01 9.835000e-01 9.800000e-01 9.735000e-01 9.795000e-01 9.595000e-01 9.845000e-01
    val_recall0.1 9.685000e-01 9.680000e-01 9.740000e-01 9.790000e-01 9.590000e-01 9.670000e-01 9.655000e-01 9.365000e-01 9.500000e-01
    val_recall0.3 9.345000e-01 9.300000e-01 9.470000e-01 9.540000e-01 9.215000e-01 9.330000e-01 9.160000e-01 8.965000e-01 9.140000e-01
    val_recall0.5 8.975000e-01 8.930000e-01 9.160000e-01 9.290000e-01 8.850000e-01 8.905000e-01 8.750000e-01 8.560000e-01 8.720000e-01
    val_recall0.7 8.445000e-01 8.380000e-01 8.685000e-01 8.890000e-01 8.310000e-01 8.450000e-01 8.215000e-01 7.985000e-01 8.145000e-01
    val_recall0.9 7.400000e-01 6.945000e-01 7.750000e-01 8.060000e-01 7.235000e-01 7.475000e-01 6.965000e-01 6.890000e-01 7.090000e-01
    val_tn0.1 1.550422e+07 1.489606e+07 1.551728e+07 1.515836e+07 1.526304e+07 7.617204e+06 1.466611e+07 1.509811e+07 3.882080e+06
    val_tn0.3 1.600058e+07 1.569691e+07 1.599514e+07 1.567769e+07 1.595088e+07 7.937553e+06 1.568612e+07 1.565312e+07 3.982077e+06
    val_tn0.5 1.619724e+07 1.600685e+07 1.620702e+07 1.587864e+07 1.618029e+07 8.075578e+06 1.601021e+07 1.591797e+07 4.034234e+06
    val_tn0.7 1.632732e+07 1.623412e+07 1.635440e+07 1.602572e+07 1.634000e+07 8.149060e+06 1.619086e+07 1.611058e+07 4.067340e+06
    val_tn0.9 1.641450e+07 1.642108e+07 1.645742e+07 1.613495e+07 1.645097e+07 8.215687e+06 1.631236e+07 1.627022e+07 4.091636e+06
    val_tp0.1 3.046124e+06 3.060968e+06 3.079936e+06 3.026346e+06 3.032762e+06 1.530771e+06 3.030885e+06 2.936426e+06 7.619985e+05
    val_tp0.3 2.939140e+06 2.940335e+06 2.995539e+06 2.949438e+06 2.914128e+06 1.476710e+06 2.874668e+06 2.810479e+06 7.332645e+05
    val_tp0.5 2.824070e+06 2.823948e+06 2.896924e+06 2.870677e+06 2.799918e+06 1.409026e+06 2.747870e+06 2.682980e+06 6.993800e+05
    val_tp0.7 2.655566e+06 2.649896e+06 2.746452e+06 2.746808e+06 2.628196e+06 1.337188e+06 2.579552e+06 2.503406e+06 6.537325e+05
    val_tp0.9 2.327260e+06 2.196472e+06 2.452454e+06 2.491253e+06 2.287070e+06 1.182887e+06 2.187046e+06 2.161117e+06 5.689300e+05

    This looks promising with a wide variety of approaches leading to success. Sadly, only run 00f2635d9fa2463c9a066722163405be had the model saved out for immediate testing. The other models have to be trained again with the same settings.

    Lets look at some plots of successful runs as well: High validation AUC: run1-2_highvalauc.png High validation AUC, high validation precision: run1-2_highvalauc-highvalprecision.png High validation AUC, high validation recall: run1-2_highvalauc-highvalrecall.png High validation AUC, high validation recall, high validation precision: run1-2_highvalauc-highvalrecall-highvalprecision.png

  5. Now lets look at some unsuccessful runs and some interesting runs: High training AUC, low validation AUC (these models overfitted). Here a clear predictor seems to be lr_power=1, a small batch size, and a small pool size. run1-2_highauc-lowvalauc.png Low validation AUC, low validation recall: run1-2_lowvalrecall-lowvalauc.png Low validation precision, but still reasonably high validation AUC: run1-2_lowvalprecision-highvalauc.png Low validation recall, but still reasonably high validation AUC: run1-2_lowvalrecall-highvalauc.png
  6. Lets inspect input sizes. Their might be a slight preference for 2**14 length of traces, but it could also be negligible. Input size of 4096 time steps run1-2_input-size_4096.png Input size of 8192 time steps run1-2_input-size_8192.png Input size of 16384 time steps run1-2_input-size_16384.png
  7. Next lets inspect a complicated hyperparameter, which adjusts the pool size of the encoder, as well as the strides and the kernel size of the decoder. It is connected to the input size and nlevels in model building, as well. The following condition has to hold: input_size >= 2 * pool_size**n_levels. Because of that, a poolsize of 8 is only possible with smaller nlevels. Still, a higher poolsize seems to be useful in this project. pool size, kernel size and strides of 2: run1-2_poolsize-strides-kernelsize_2.png pool size, kernel size and strides of 4: run1-2_poolsize-strides-kernelsize_4.png pool size, kernel size and strides of 8: run1-2_poolsize-strides-kernelsize_8.png
  8. Another complicated hyperparameter is nlevels. It refers to the "depth" of the unet. How many times will the input trace be pooled and skip connections made? This parameter strongly influences the amount of total model parameters and smaller values make the model much more portable and "simpler", which could be viewed as a positive thing itself. A larger value raises the learning capacity of the network, but also raises the possibility of overfitting. Number of unet levels 1 to 3 run1-2_n-levels_1-3.png Number of unet levels 4 to 6 run1-2_n-levels_4-6.png Number of unet levels 7 to 9 run1-2_n-levels_7-9.png
  9. Lets take a look at the different first filters (starting from this value, the number of filters was doubled for each unet level till a maximum of 512 filters was reached). More filters mean more capacity for learning, but also more risk of overfitting and model size. Here, no real trend is visible, so a lower number of filters should be sufficient. from 6 to 45 first filters run1-2_first-filters_6-45.png from 57 to 77 first filters run1-2_first-filters_57-77.png from 100 to 128 first filters run1-2_first-filters_100-128.png
  10. Now lets take a look at the starting learning rates and the learning rate power, which together are used to construct a learning rate schedule. For the learning rate starting value, the lower category seemed to perform better than the upper category. The power of 1 is simpler than the values between 4 and 7. Start learning rate between 0.0065 and 0.035: run1-2_lr-start_0.0065-0.035.png Start learning rate between 0.045 and 0.065: run1-2_lr-start_0.045-0.065.png Start learning rate between 0.065 and 0.1: run1-2_lr-start_0.065-0.1.png Power of learning rate equation = 1, meaning a linear decay: run1-2_lr-power_1.png Power of learning rate equation = 4 to 7, meaning a polynomial decay: run1-2_lr-power_2-7.png
  11. The next parameter is the batch size, which is directly coupled to the steps per epoch of the training (since I chose the step size to equal the total number of examples divided by the batch size). A larger batch size pools more training examples for decisions. We see that batch size >7 seem to be benefitial. Batch size of 2 to 8, with steps per epoch between 600 and 2400 run1-2_batch-size_2-8_higher-steps-per-epoch.png Batch size of 8 to 20, with steps per epoch between 165 and 550 run1-2_batch-size_8-20_lower-steps-per-epoch.png
  12. Lastly, lets take a look at the different scalers. I used a scaler from the beginning, because in learning algorithms this guarantees some numerical stability. As a naive approach I chose one - the Min-Max scaler. After evaluating the first training approaches I noticed that there might be problems with this scaler, if the trace has no artifacts in it, see this example plot, where I plotted a variety of scalers from sklearn on some simulated data: scalers_simulations.png From these I chose Standard, Robust, Max-Abs, Quantile (Gaussian), Min-Max, L1 and L2, because they looked most promising. Now, let's load some of the experimental data and see how these scalers look on this data.
    path_tb_pex5_egfp = '/beegfs/ye53nis/data/Pablo_structured_experiment'
    length_delimiter = 2**13  # for U-Net
    bin_for_correlation = 1e6
    ptu_1ms, _ = ptu.import_from_ptu(
          path=path_tb_pex5_egfp,
          file_delimiter=2,
          photon_count_bin=1e6,
          verbose=True)
    ptu_1ms = ptu_1ms
    
    1 of 2: /beegfs/ye53nis/data/Pablo_structured_experiment/DiO LUV 10uM in 20 nM AF48822_T273s_1.ptu
    2 of 2: /beegfs/ye53nis/data/Pablo_structured_experiment/DiO LUV 10uM in 20 nM AF48821_T260s_1.ptu
    
    scalers = {'Unscaled data': None,
               'Data after standard scaling (z-score)': 'standard',
               'Data after robust scaling': 'robust',
               'Data after max-abs scaling': 'maxabs',
               'Data after quantile transformation (gaussian pdf)': 'quant_g',
               'Data after min-max scaling': 'minmax',
               'Data after sample-wise L1 normalization (taxicab, LASSO)': 'l1',
               'Data after sample-wise L2 normalization (Euclidian)': 'l2'}
    plt.figure(figsize=(16,14), facecolor='white')
    for i, (text, s) in enumerate(scalers.items()):
        plt.subplot(3, 3, i+1, title=text)
        if s is None:
            plt.plot(ptu_1ms, alpha=0.75)
        else:
            print(np.array(ptu_1ms.iloc[:, 0]))
            # We have to circumvent the following error because we want to dropna() for scaling:
            # /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/numpy/core/_asarray.py:136:
            # VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which
            # is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes)
            # is deprecated. If you meant to do this, you must specify 'dtype=object' when creating
            # the ndarray: return array(a, dtype, copy=False, order=order, subok=True)
            # And the sklearn error:
            # ValueError: Expected 2D array, got 1D array instead:
            # array=[486. 455. 622. ... 580. 522. 682.].
            # Reshape your data either using array.reshape(-1, 1) if your data has a single feature
            # or array.reshape(1, -1) if it contains a single sample.
            tscaled = [ppd.scale_trace(np.array(ptu_1ms.iloc[:, j].dropna(), dtype=object).reshape(-1, 1),
                                       scaler=s) for j in range(2)]
            [plt.plot(tscaled[k], alpha=0.75) for k in range(2)]
    plt.tight_layout()
    
    

    with reshape(-1, 1) to say that data has a single feature scalers_experimental.png . We already see that the quantile transformation might not be so useful here. In the simulated data with slower cluster speeds it made the peaks clearly distinct, here this is not the case.

    using reshape(1, -1) to say that data contains a single sample does not work e.g. each timestep normalized to 0…

    First, let's look at L1 and L2. L1 was only chosen once, L2 got chosen more often. In runs with the highest val_auc, this norm especially seems to lead to a high recall (which is a problem with minmax). But we also see that depending on the other parameters there is a wide variation of results, including very bad ones run1-2_scaler_l1.png run1-2_scaler_l2.png

    Second, let's look at Max-Abs (from 0 to the maximum value) and Min-Max (from the minimum to the maximum value). Both are conceptually somewhat similar, although Max-Abs keeps the distance from 0 to the minimal value, and Min-Max does not. We see that both lead to good results, with some variation. In Max-Abs, the spread seems to be wider. run1-2_scaler_maxabs.png run1-2_scaler_minmax.png

    Third, Quantile Transformation (Gaussian distribution). We see that this transformation suffers from bad valprecision values. Higher recall values can be reached with other transformations. Thus, I think I can neglect this transformation in the future. run1-2_scaler_quant-g.png

    Fourth, the Robust Scaler. It robustly (haha) achieves good precision and recall values run1-2_scaler_robust.png

    Fifth, the Standard Scaler. It achieves some of the best results, even though there is some variation, both in precision and recall. run1-2_scaler_standard.png

2.4.5 Look at training logging of training run 00f2635d9fa2463c9a066722163405be

This run is one of the few very good runs where the model got saved out.

  1. Let's look at some prediction plots after 1 epoch: plot0.png after 5 epochs: plot4.png after 10 epochs: plot9.png after 15 epochs: plot14.png after 20 epochs: plot19.png
  2. Now let's print out the model architecture. -n prints out line numbers.
    cat -n data/mlruns/8/00f2635d9fa2463c9a066722163405be/artifacts/model_summary.txt
    
    |  1 | unet_depth5                                                                                        |
    |  2 | __________________________________________________________________________________________________ |
    |  3 | Layer (type)                    Output Shape         Param #     Connected to                      |
    |  4 | ================================================================================================== |
    |  5 | input_6 (InputLayer)            [(None, 16384, 1)]   0                                             |
    |  6 | __________________________________________________________________________________________________ |
    |  7 | encode0 (Sequential)            (None, 16384, 6)     186         input_6[0][0]                     |
    |  8 | __________________________________________________________________________________________________ |
    |  9 | mp_encode0 (MaxPooling1D)       (None, 4096, 6)      0           encode0[0][0]                     |
    | 10 | __________________________________________________________________________________________________ |
    | 11 | encode1 (Sequential)            (None, 4096, 12)     768         mp_encode0[0][0]                  |
    | 12 | __________________________________________________________________________________________________ |
    | 13 | mp_encode1 (MaxPooling1D)       (None, 1024, 12)     0           encode1[0][0]                     |
    | 14 | __________________________________________________________________________________________________ |
    | 15 | encode2 (Sequential)            (None, 1024, 24)     2832        mp_encode1[0][0]                  |
    | 16 | __________________________________________________________________________________________________ |
    | 17 | mp_encode2 (MaxPooling1D)       (None, 256, 24)      0           encode2[0][0]                     |
    | 18 | __________________________________________________________________________________________________ |
    | 19 | encode3 (Sequential)            (None, 256, 48)      10848       mp_encode2[0][0]                  |
    | 20 | __________________________________________________________________________________________________ |
    | 21 | mp_encode3 (MaxPooling1D)       (None, 64, 48)       0           encode3[0][0]                     |
    | 22 | __________________________________________________________________________________________________ |
    | 23 | encode4 (Sequential)            (None, 64, 96)       42432       mp_encode3[0][0]                  |
    | 24 | __________________________________________________________________________________________________ |
    | 25 | mp_encode4 (MaxPooling1D)       (None, 16, 96)       0           encode4[0][0]                     |
    | 26 | __________________________________________________________________________________________________ |
    | 27 | two_conv_center (Sequential)    (None, 16, 192)      167808      mp_encode4[0][0]                  |
    | 28 | __________________________________________________________________________________________________ |
    | 29 | conv_transpose_decoder4 (Sequen (None, 64, 192)      148416      two_conv_center[0][0]             |
    | 30 | __________________________________________________________________________________________________ |
    | 31 | decoder4 (Concatenate)          (None, 64, 288)      0           encode4[0][0]                     |
    | 32 | conv_transpose_decoder4[0][0]                                                                      |
    | 33 | __________________________________________________________________________________________________ |
    | 34 | two_conv_decoder4 (Sequential)  (None, 64, 192)      278400      decoder4[0][0]                    |
    | 35 | __________________________________________________________________________________________________ |
    | 36 | conv_transpose_decoder3 (Sequen (None, 256, 96)      74208       two_conv_decoder4[0][0]           |
    | 37 | __________________________________________________________________________________________________ |
    | 38 | decoder3 (Concatenate)          (None, 256, 144)     0           encode3[0][0]                     |
    | 39 | conv_transpose_decoder3[0][0]                                                                      |
    | 40 | __________________________________________________________________________________________________ |
    | 41 | two_conv_decoder3 (Sequential)  (None, 256, 96)      70080       decoder3[0][0]                    |
    | 42 | __________________________________________________________________________________________________ |
    | 43 | conv_transpose_decoder2 (Sequen (None, 1024, 48)     18672       two_conv_decoder3[0][0]           |
    | 44 | __________________________________________________________________________________________________ |
    | 45 | decoder2 (Concatenate)          (None, 1024, 72)     0           encode2[0][0]                     |
    | 46 | conv_transpose_decoder2[0][0]                                                                      |
    | 47 | __________________________________________________________________________________________________ |
    | 48 | two_conv_decoder2 (Sequential)  (None, 1024, 48)     17760       decoder2[0][0]                    |
    | 49 | __________________________________________________________________________________________________ |
    | 50 | conv_transpose_decoder1 (Sequen (None, 4096, 24)     4728        two_conv_decoder2[0][0]           |
    | 51 | __________________________________________________________________________________________________ |
    | 52 | decoder1 (Concatenate)          (None, 4096, 36)     0           encode1[0][0]                     |
    | 53 | conv_transpose_decoder1[0][0]                                                                      |
    | 54 | __________________________________________________________________________________________________ |
    | 55 | two_conv_decoder1 (Sequential)  (None, 4096, 24)     4560        decoder1[0][0]                    |
    | 56 | __________________________________________________________________________________________________ |
    | 57 | conv_transpose_decoder0 (Sequen (None, 16384, 12)    1212        two_conv_decoder1[0][0]           |
    | 58 | __________________________________________________________________________________________________ |
    | 59 | decoder0 (Concatenate)          (None, 16384, 18)    0           encode0[0][0]                     |
    | 60 | conv_transpose_decoder0[0][0]                                                                      |
    | 61 | __________________________________________________________________________________________________ |
    | 62 | two_conv_decoder0 (Sequential)  (None, 16384, 12)    1200        decoder0[0][0]                    |
    | 63 | __________________________________________________________________________________________________ |
    | 64 | conv1d_89 (Conv1D)              (None, 16384, 1)     13          two_conv_decoder0[0][0]           |
    | 65 | ================================================================================================== |
    | 66 | Total params: 844,123                                                                              |
    | 67 | Trainable params: 840,379                                                                          |
    | 68 | Non-trainable params: 3,744                                                                        |
    | 69 | __________________________________________________________________________________________________ |
    
  3. Print out mlflow Model parameters
    cat -n data/mlruns/8/00f2635d9fa2463c9a066722163405be/artifacts/model/MLmodel
    
    |  1 | artifact_path: model                           |
    |  2 | flavors:                                       |
    |  3 | keras:                                         |
    |  4 | data: data                                     |
    |  5 | keras_module: tensorflow.keras                 |
    |  6 | keras_version: 2.5.0                           |
    |  7 | save_format: tf                                |
    |  8 | python_function:                               |
    |  9 | data: data                                     |
    | 10 | env: conda.yaml                                |
    | 11 | loader_module: mlflow.keras                    |
    | 12 | python_version: 3.9.6                          |
    | 13 | run_id: 00f2635d9fa2463c9a066722163405be       |
    | 14 | utc_time_created: '2021-08-08 17:14:06.771348' |
    
  4. Print out training data parameters. One learning: The f1-metric does not work!
    echo "time val_recall0.5 epoch"
    cat data/mlruns/8/00f2635d9fa2463c9a066722163405be/metrics/val_recall0.5
    echo ""
    echo "time val_precision0.5 epoch"
    cat data/mlruns/8/00f2635d9fa2463c9a066722163405be/metrics/val_precision0.5
    echo ""
    echo "time val_auc epoch"
    cat data/mlruns/8/00f2635d9fa2463c9a066722163405be/metrics/val_auc
    echo ""
    echo "time val_f1 epoch"
    cat data/mlruns/8/00f2635d9fa2463c9a066722163405be/metrics/val_f1
    
    time val\recall0.5 epoch
    1628442521951 1.0 0
    1628442538552 1.0 1
    1628442555029 0.9884297847747803 2
    1628442571921 0.9939945340156555 3
    1628442588629 0.9871219396591187 4
    1628442605537 0.9595196843147278 5
    1628442622450 0.9733913540840149 6
    1628442642363 0.9914226531982422 7
    1628442659452 0.9950642585754395 8
    1628442676492 0.7978640794754028 9
    1628442693576 0.9715087413787842 10
    1628442710632 0.9640132188796997 11
    1628442727806 0.9724437594413757 12
    1628442744594 0.9310197234153748 13
    1628442761707 0.8998530507087708 14
    1628442778890 0.9342276453971863 15
    1628442795829 0.9401260018348694 16
    1628442812832 0.9072213768959045 17
    1628442829592 0.8626136183738708 18
    1628442846600 0.9356982707977295 19
         
    time val\precision0.5 epoch
    1628442521951 0.161118745803833 0
    1628442538552 0.16019564867019653 1
    1628442555029 0.17915846407413483 2
    1628442571921 0.243782639503479 3
    1628442588629 0.2357945740222931 4
    1628442605537 0.24214163422584534 5
    1628442622450 0.5458978414535522 6
    1628442642363 0.4380616247653961 7
    1628442659452 0.2929491698741913 8
    1628442676492 0.8729878067970276 9
    1628442693576 0.7323421835899353 10
    1628442710632 0.5339199304580688 11
    1628442727806 0.7172960042953491 12
    1628442744594 0.8843752145767212 13
    1628442761707 0.9003584980964661 14
    1628442778890 0.8853111267089844 15
    1628442795829 0.8726167678833008 16
    1628442812832 0.9290499091148376 17
    1628442829592 0.9494153261184692 18
    1628442846600 0.8976160287857056 19
         
    time val\auc epoch
    1628442521951 0.5 0
    1628442538552 0.5000137686729431 1
    1628442555029 0.5814657211303711 2
    1628442571921 0.8558493852615356 3
    1628442588629 0.7804751396179199 4
    1628442605537 0.7518471479415894 5
    1628442622450 0.9729548096656799 6
    1628442642363 0.980221688747406 7
    1628442659452 0.9016830325126648 8
    1628442676492 0.9798668026924133 9
    1628442693576 0.9884295463562012 10
    1628442710632 0.9497971534729004 11
    1628442727806 0.9887964129447937 12
    1628442744594 0.9877431392669678 13
    1628442761707 0.9804221987724304 14
    1628442778890 0.9873037338256836 15
    1628442795829 0.9889394640922546 16
    1628442812832 0.9840959310531616 17
    1628442829592 0.9767916202545166 18
    1628442846600 0.9875953793525696 19
         
    time val\f1 epoch
    1628442521951 0.2775232493877411 0
    1628442538552 0.2761525511741638 1
    1628442555029 0.27699539065361023 2
    1628442571921 0.27680331468582153 3
    1628442588629 0.27761173248291016 4
    1628442605537 0.2781853973865509 5
    1628442622450 0.27713820338249207 6
    1628442642363 0.27754780650138855 7
    1628442659452 0.27742522954940796 8
    1628442676492 0.2759542465209961 9
    1628442693576 0.27791914343833923 10
    1628442710632 0.2769533693790436 11
    1628442727806 0.2773088812828064 12
    1628442744594 0.2751254439353943 13
    1628442761707 0.2785097658634186 14
    1628442778890 0.27688315510749817 15
    1628442795829 0.2764797508716583 16
    1628442812832 0.27767083048820496 17
    1628442829592 0.2765066623687744 18
    1628442846600 0.27736371755599976 19
  5. Of course I have plotted out a lot of further info (tensorflow, mlflow, experiment parameters, etc). But I will plot this out in the dedicated training run, not in the hparams run.

2.4.6 Apply model from run 2 to experimental data

%cd /beegfs/ye53nis/drmed-git
/beegfs/ye53nis/drmed-git
from pathlib import Path

import sys
import mlflow
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import tensorflow as tf
print("tf version: ", tf.version.VERSION)
print("tf.keras version: ", tf.keras.__version__)
print("mlflow version: ", mlflow.version.VERSION)
tf version:  2.5.0
tf.keras version:  2.5.0
mlflow version:  1.19.0
sys.path.append('src/')
from fluotracify.simulations import (
   import_simulation_from_csv as isfc,
   analyze_simulations as ans,
)
from fluotracify.training import build_model as bm, preprocess_data as ppd
from fluotracify.applications import correlate, plots, correction
from fluotracify.imports import ptu_utils as ptu
import importlib
importlib.reload(correction)
logged_model = 'file:///beegfs/ye53nis/drmed-git/data/mlruns/8/00f2635d9fa2463c9a066722163405be/artifacts/model'
metrics_thresholds = [0.1, 0.3, 0.5, 0.7, 0.9]

loaded_model = mlflow.keras.load_model(logged_model, compile=False)
loaded_model.compile(loss=bm.binary_ce_dice_loss(),
                     optimizer=tf.keras.optimizers.Adam(),
                     metrics = bm.unet_metrics(metrics_thresholds))
loaded_model
WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
<tensorflow.python.keras.engine.functional.Functional at 0x2b21c027e8b0>
# The following block has to be executed multiple times with different array lengths
# e.g. 2**14, 2**13, 2**12
# note that: after executing 2**14, then 2**13, there will be an error
# if you then execute 2**12 and then 2**13 again, the error will be gone
test_features = np.zeros((2**14)).reshape(1, -1, 1)
print(test_features.shape)
predictions = loaded_model.predict(test_features, verbose=0).flatten()
predictions
(1, 16384, 1)
array([1.4064237e-01, 1.3650321e-08, 7.4799689e-10, ..., 3.1755112e-10,
       1.8236226e-10, 6.6588524e-10], dtype=float32)
test_features2 = np.zeros((2**13)).reshape(1, -1, 1)
print(test_features2.shape)
predictions = loaded_model.predict(test_features2, verbose=0).flatten()
predictions
(1, 8192, 1)
array([1.4064237e-01, 1.3650321e-08, 7.4799689e-10, ..., 3.1755112e-10,
       1.8236226e-10, 6.6588524e-10], dtype=float32)
path_tb_pex5_egfp = '/beegfs/ye53nis/data/Pablo_structured_experiment'
pred_thresh = [0.1, 0.3, 0.5, 0.7, 0.9]
length_delimiter = 2**13  # for U-Net
bin_for_correlation = 1e6
out = correction.correct_experimental_traces_from_ptu_by_unet_prediction(
  path_list=path_tb_pex5_egfp,
  model=loaded_model,
  pred_thresh=pred_thresh,
  photon_count_bin=bin_for_correlation,
  ntraces=None,
  save_as_csv=True)
out
Loading dataset 1 from path /beegfs/ye53nis/data/Pablo_structured_experiment with bin=1e6. This can take a while...
1 of 2: /beegfs/ye53nis/data/Pablo_structured_experiment/DiO LUV 10uM in 20 nM AF48822_T273s_1.ptu
2 of 2: /beegfs/ye53nis/data/Pablo_structured_experiment/DiO LUV 10uM in 20 nM AF48821_T260s_1.ptu
Processing correlation of unprocessed dataset 1
Processing correlation with correction by prediction of dataset 1
  \(D\) in \(\frac{{\mu m^2}}{{s}}\) \(\tau_{{D}}\) in \(ms\) Trace lengths folderid-tracesused Photon count bin for correlation in \(ns\) FileGUID FileCreatingTime MeasurementSubMode FileComment TTResultStopReason MeasDescGlobalResolution TTResultNumberOfRecords MeasDescAcquisitionTime TTResultMDescWarningFlags TTResultStopAfter TTResultFormatTTTRRecType TTResultFormatBitsPerRecord UsrPowerDiode HeaderEnd Number of Channels
0 0.898516 12.544077 8192 0-orig 1000000.0 {1FADBD63-E853-4886-82EB-0D4FEB0A2641} (2019, 10, 31, 11, 17, 29, 3, 304, 0) 0   0 0.0 6124808 10000 0 10000 16843524 32 1002.253351 <empty Tag> 1
1 1.488758 7.570775 8192 0-orig 1000000.0 {1104778F-509F-40AB-9176-580B3A1CE38D} (2019, 10, 31, 11, 17, 16, 3, 304, 0) 0   0 0.0 5881858 10000 0 10000 16843524 32 979.946455 <empty Tag> 1
2 7.747958 1.454713 5053 0-pred-0.1 1000000.0 {1FADBD63-E853-4886-82EB-0D4FEB0A2641} (2019, 10, 31, 11, 17, 29, 3, 304, 0) 0   0 0.0 6124808 10000 0 10000 16843524 32 1002.253351 <empty Tag> 1
3 2.388373 4.719135 5926 0-pred-0.1 1000000.0 {1104778F-509F-40AB-9176-580B3A1CE38D} (2019, 10, 31, 11, 17, 16, 3, 304, 0) 0   0 0.0 5881858 10000 0 10000 16843524 32 979.946455 <empty Tag> 1
4 5.875199 1.918413 5729 0-pred-0.3 1000000.0 {1FADBD63-E853-4886-82EB-0D4FEB0A2641} (2019, 10, 31, 11, 17, 29, 3, 304, 0) 0   0 0.0 6124808 10000 0 10000 16843524 32 1002.253351 <empty Tag> 1
5 2.144527 5.255731 6707 0-pred-0.3 1000000.0 {1104778F-509F-40AB-9176-580B3A1CE38D} (2019, 10, 31, 11, 17, 16, 3, 304, 0) 0   0 0.0 5881858 10000 0 10000 16843524 32 979.946455 <empty Tag> 1
6 3.673109 3.068532 6179 0-pred-0.5 1000000.0 {1FADBD63-E853-4886-82EB-0D4FEB0A2641} (2019, 10, 31, 11, 17, 29, 3, 304, 0) 0   0 0.0 6124808 10000 0 10000 16843524 32 1002.253351 <empty Tag> 1
7 1.816449 6.204994 7164 0-pred-0.5 1000000.0 {1104778F-509F-40AB-9176-580B3A1CE38D} (2019, 10, 31, 11, 17, 16, 3, 304, 0) 0   0 0.0 5881858 10000 0 10000 16843524 32 979.946455 <empty Tag> 1
8 1.857476 6.067942 6576 0-pred-0.7 1000000.0 {1FADBD63-E853-4886-82EB-0D4FEB0A2641} (2019, 10, 31, 11, 17, 29, 3, 304, 0) 0   0 0.0 6124808 10000 0 10000 16843524 32 1002.253351 <empty Tag> 1
9 1.45624 7.739836 7446 0-pred-0.7 1000000.0 {1104778F-509F-40AB-9176-580B3A1CE38D} (2019, 10, 31, 11, 17, 16, 3, 304, 0) 0   0 0.0 5881858 10000 0 10000 16843524 32 979.946455 <empty Tag> 1
10 0.986952 11.420058 7081 0-pred-0.9 1000000.0 {1FADBD63-E853-4886-82EB-0D4FEB0A2641} (2019, 10, 31, 11, 17, 29, 3, 304, 0) 0   0 0.0 6124808 10000 0 10000 16843524 32 1002.253351 <empty Tag> 1
11 1.150654 9.795347 7735 0-pred-0.9 1000000.0 {1104778F-509F-40AB-9176-580B3A1CE38D} (2019, 10, 31, 11, 17, 16, 3, 304, 0) 0   0 0.0 5881858 10000 0 10000 16843524 32 979.946455 <empty Tag> 1

12 rows × 94 columns

path_pex5_exp = ['/beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu', '/beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu']
pred_thresh = [0.1, 0.3, 0.5, 0.7, 0.9]
length_delimiter = 2**13  # for U-Net
bin_for_correlation = 1e5
out = correction.correct_experimental_traces_from_ptu_by_unet_prediction(
  path_list=path_pex5_exp,
  model=loaded_model,
  pred_thresh=pred_thresh,
  photon_count_bin=bin_for_correlation,
  ntraces=400,
  save_as_csv=True)
out
Loading dataset 1 from path /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu with bin=1e6. This can take a while...
1 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48891_T1082s_1.ptu
2 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488129_T1537s_1.ptu
3 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488210_T2505s_1.ptu
4 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488199_T2375s_1.ptu
5 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488171_T2040s_1.ptu
6 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488229_T2733s_1.ptu
7 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48890_T1070s_1.ptu
8 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488113_T1352s_1.ptu
9 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48883_T986s_1.ptu10 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488227_T2709s_1.ptu
11 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488174_T2088s_1.ptu
12 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488102_T1214s_1.ptu
13 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488131_T1561s_1.ptu
14 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48826_T302s_1.ptu
15 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488162_T1932s_1.ptu
16 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488190_T2267s_1.ptu
17 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488189_T2255s_1.ptu
18 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488208_T2481s_1.ptu
19 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48827_T314s_1.ptu
20 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488221_T2637s_1.ptu
21 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488153_T1834s_1.ptu
22 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488157_T1883s_1.ptu
23 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488197_T2351s_1.ptu
24 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48822_T254s_1.ptu
25 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48816_T182s_1.ptu
26 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488179_T2148s_1.ptu
27 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488104_T1238s_1.ptu
28 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48870_T833s_1.ptu
29 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48857_T676s_1.ptu
30 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488125_T1496s_1.ptu
31 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48839_T459s_1.ptu
32 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488135_T1610s_1.ptu
33 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48850_T592s_1.ptu
34 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488159_T1907s_1.ptu
35 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48851_T604s_1.ptu
36 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488158_T1895s_1.ptu
37 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48878_T926s_1.ptu
38 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48838_T447s_1.ptu
39 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48869_T821s_1.ptu
40 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48834_T399s_1.ptu
41 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48843_T506s_1.ptu
42 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48890_T1076s_1.ptu
43 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488184_T2209s_1.ptu
44 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48863_T748s_1.ptu
45 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488103_T1232s_1.ptu
46 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488181_T2159s_1.ptu
47 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48859_T698s_1.ptu
48 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48872_T855s_1.ptu
49 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488150_T1789s_1.ptu
50 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488241_T2877s_1.ptu
51 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488204_T2434s_1.ptu
52 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48877_T918s_1.ptu
53 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488180_T2147s_1.ptu
54 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488183_T2197s_1.ptu
55 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488160_T1908s_1.ptu
56 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488238_T2842s_1.ptu
57 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488165_T1968s_1.ptu
58 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488234_T2793s_1.ptu
59 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48882_T979s_1.ptu
60 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48896_T1148s_1.ptu
61 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488107_T1280s_1.ptu
62 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488138_T1654s_1.ptu
63 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488124_T1484s_1.ptu
64 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488187_T2245s_1.ptu
65 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48874_T879s_1.ptu
66 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488159_T1896s_1.ptu
67 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488177_T2124s_1.ptu
68 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488111_T1328s_1.ptu
69 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48873_T867s_1.ptu
70 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48881_T962s_1.ptu
71 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48853_T628s_1.ptu
72 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48842_T496s_1.ptu
73 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48850_T590s_1.ptu
74 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48899_T1184s_1.ptu
75 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488196_T2339s_1.ptu
76 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488164_T1956s_1.ptu
77 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488217_T2589s_1.ptu
78 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488218_T2601s_1.ptu
79 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488119_T1418s_1.ptu
80 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488206_T2458s_1.ptu
81 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48866_T785s_1.ptu
82 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488103_T1226s_1.ptu
83 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48824_T278s_1.ptu
84 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4887_T74s_1.ptu
85 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488176_T2099s_1.ptu
86 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48817_T194s_1.ptu
87 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488113_T1346s_1.ptu
88 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48858_T688s_1.ptu
89 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488211_T2517s_1.ptu
90 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48860_T712s_1.ptu
91 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48833_T387s_1.ptu
92 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48858_T686s_1.ptu
93 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48813_T146s_1.ptu
94 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488126_T1502s_1.ptu
95 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48894_T1118s_1.ptu
96 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488110_T1316s_1.ptu
97 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488188_T2243s_1.ptu
98 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488120_T1430s_1.ptu
99 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488219_T2613s_1.ptu
100 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488138_T1646s_1.ptu
101 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488187_T2231s_1.ptu
102 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488146_T1742s_1.ptu
103 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488155_T1859s_1.ptu
104 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488174_T2075s_1.ptu
105 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488158_T1884s_1.ptu
106 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488244_T2914s_1.ptu
107 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4886_T62s_1.ptu
108 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48885_T1015s_1.ptu
109 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488165_T1979s_1.ptu
110 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48857_T674s_1.ptu
111 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488191_T2279s_1.ptu
112 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488183_T2183s_1.ptu
113 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488141_T1689s_1.ptu
114 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488189_T2269s_1.ptu
115 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48823_T266s_1.ptu
116 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488209_T2493s_1.ptu
117 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488136_T1630s_1.ptu
118 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488185_T2221s_1.ptu
119 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488179_T2135s_1.ptu
120 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48866_T782s_1.ptu
121 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488123_T1472s_1.ptu
122 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488160_T1919s_1.ptu
123 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48856_T664s_1.ptu
124 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48830_T351s_1.ptu
125 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48832_T375s_1.ptu
126 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48859_T700s_1.ptu
127 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488129_T1545s_1.ptu
128 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488143_T1714s_1.ptu
129 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48886_T1022s_1.ptu
130 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488147_T1754s_1.ptu
131 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488193_T2303s_1.ptu
132 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488105_T1250s_1.ptu
133 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488228_T2721s_1.ptu
134 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48891_T1088s_1.ptu
135 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488170_T2040s_1.ptu
136 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48835_T411s_1.ptu
137 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48888_T1046s_1.ptu
138 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488123_T1466s_1.ptu
139 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48876_T906s_1.ptu
140 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48822_T255s_1.ptu
141 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488145_T1730s_1.ptu
142 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488114_T1358s_1.ptu
143 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48824_T279s_1.ptu
144 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48854_T638s_1.ptu
145 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488247_T2949s_1.ptu
146 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48844_T519s_1.ptu
147 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48841_T482s_1.ptu
148 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48828_T327s_1.ptu
149 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48840_T471s_1.ptu
150 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488225_T2685s_1.ptu
151 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48831_T362s_1.ptu
152 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48849_T578s_1.ptu
153 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48861_T722s_1.ptu
154 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48852_T614s_1.ptu
155 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48852_T616s_1.ptu
156 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488149_T1778s_1.ptu
157 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488169_T2028s_1.ptu
158 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4883_T26s_1.ptu
159 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48881_T966s_1.ptu
160 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48837_T435s_1.ptu
161 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488139_T1666s_1.ptu
162 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488194_T2315s_1.ptu
163 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488205_T2446s_1.ptu
164 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48818_T206s_1.ptu
165 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488128_T1533s_1.ptu
166 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48851_T602s_1.ptu
167 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488116_T1388s_1.ptu
168 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488184_T2196s_1.ptu
169 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488117_T1394s_1.ptu
170 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48869_T818s_1.ptu
171 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48895_T1130s_1.ptu
172 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488173_T2076s_1.ptu
173 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488106_T1262s_1.ptu
174 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488249_T2993s_1.ptu
175 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48889_T1063s_1.ptu
176 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48845_T531s_1.ptu
177 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488117_T1400s_1.ptu
178 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488192_T2291s_1.ptu
179 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488166_T1992s_1.ptu
180 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48896_T1142s_1.ptu
181 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48894_T1124s_1.ptu
182 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488177_T2111s_1.ptu
183 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48862_T736s_1.ptu
184 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488118_T1412s_1.ptu
185 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48892_T1100s_1.ptu
186 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48847_T555s_1.ptu
187 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488223_T2661s_1.ptu
188 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48853_T626s_1.ptu
189 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488136_T1622s_1.ptu
190 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488151_T1810s_1.ptu
191 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488245_T2925s_1.ptu
192 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48879_T942s_1.ptu
193 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488121_T1448s_1.ptu
194 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48868_T809s_1.ptu
195 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488201_T2398s_1.ptu
196 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488140_T1670s_1.ptu
197 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488250_T3006s_1.ptu
198 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4884_T38s_1.ptu
199 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488127_T1514s_1.ptu
200 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488104_T1244s_1.ptu
201 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488181_T2172s_1.ptu
202 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488115_T1370s_1.ptu
203 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488140_T1678s_1.ptu
204 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48855_T652s_1.ptu
205 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48884_T1003s_1.ptu
206 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48846_T544s_1.ptu
207 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488131_T1569s_1.ptu
208 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488236_T2817s_1.ptu
209 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488178_T2123s_1.ptu
210 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48873_T869s_1.ptu
211 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488164_T1968s_1.ptu
212 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488249_T2973s_1.ptu
213 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488151_T1801s_1.ptu
214 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488235_T2805s_1.ptu
215 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488170_T2028s_1.ptu
216 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488180_T2160s_1.ptu
217 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488135_T1618s_1.ptu
218 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488190_T2281s_1.ptu
219 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48864_T758s_1.ptu
220 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48867_T794s_1.ptu
221 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48875_T891s_1.ptu
222 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48871_T842s_1.ptu
223 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488213_T2541s_1.ptu
224 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488182_T2184s_1.ptu
225 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48841_T484s_1.ptu
226 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488171_T2052s_1.ptu
227 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488153_T1825s_1.ptu
228 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488132_T1581s_1.ptu
229 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488147_T1762s_1.ptu
230 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488110_T1310s_1.ptu
231 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48885_T1010s_1.ptu
232 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48825_T290s_1.ptu
233 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488168_T2004s_1.ptu
234 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48820_T230s_1.ptu
235 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488162_T1944s_1.ptu
236 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488203_T2422s_1.ptu
237 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488152_T1822s_1.ptu
238 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48821_T243s_1.ptu
239 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488167_T2004s_1.ptu
240 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48837_T436s_1.ptu
241 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488166_T1980s_1.ptu
242 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488200_T2386s_1.ptu
243 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488122_T1454s_1.ptu
244 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48843_T508s_1.ptu
245 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488102_T1220s_1.ptu
246 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48865_T773s_1.ptu
247 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488161_T1920s_1.ptu
248 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48823_T267s_1.ptu
249 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48863_T746s_1.ptu
250 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488132_T1573s_1.ptu
251 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48867_T797s_1.ptu
252 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488144_T1726s_1.ptu
253 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488145_T1738s_1.ptu
254 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488169_T2016s_1.ptu
255 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48889_T1058s_1.ptu
256 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48860_T710s_1.ptu
257 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48831_T363s_1.ptu
258 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488215_T2565s_1.ptu
259 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48878_T930s_1.ptu
260 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48865_T770s_1.ptu
261 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48870_T830s_1.ptu
262 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488114_T1364s_1.ptu
263 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488148_T1774s_1.ptu
264 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488224_T2673s_1.ptu
265 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488172_T2052s_1.ptu
266 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488167_T1992s_1.ptu
267 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488100_T1190s_1.ptu
268 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48821_T242s_1.ptu
269 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488240_T2865s_1.ptu
270 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488142_T1701s_1.ptu
271 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488130_T1549s_1.ptu
272 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48844_T520s_1.ptu
273 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48861_T724s_1.ptu
274 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48814_T158s_1.ptu
275 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488163_T1956s_1.ptu
276 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488173_T2064s_1.ptu
277 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4885_T50s_1.ptu
278 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48849_T580s_1.ptu
279 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48840_T472s_1.ptu
280 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48883_T991s_1.ptu
281 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488247_T2969s_1.ptu
282 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488150_T1798s_1.ptu
283 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48848_T566s_1.ptu
284 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48897_T1160s_1.ptu
285 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488118_T1406s_1.ptu
286 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4889_T98s_1.ptu
287 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48812_T134s_1.ptu
288 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488222_T2649s_1.ptu
289 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48887_T1039s_1.ptu
290 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48879_T938s_1.ptu
291 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48888_T1051s_1.ptu
292 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48876_T902s_1.ptu
293 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48819_T218s_1.ptu
294 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48893_T1112s_1.ptu
295 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488121_T1442s_1.ptu
296 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488101_T1208s_1.ptu
297 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488109_T1304s_1.ptu
298 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488128_T1526s_1.ptu
299 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488207_T2470s_1.ptu
300 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488246_T2937s_1.ptu
301 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488242_T2890s_1.ptu
302 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48854_T640s_1.ptu
303 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48847_T554s_1.ptu
304 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488108_T1292s_1.ptu
305 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488134_T1597s_1.ptu
306 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4888_T86s_1.ptu
307 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48872_T857s_1.ptu
308 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488149_T1786s_1.ptu
309 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48825_T291s_1.ptu
310 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488182_T2171s_1.ptu
311 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488156_T1871s_1.ptu
312 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488220_T2624s_1.ptu
313 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488248_T2981s_1.ptu
314 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48897_T1154s_1.ptu
315 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48848_T568s_1.ptu
316 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488175_T2100s_1.ptu
317 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488154_T1847s_1.ptu
318 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488245_T2945s_1.ptu
319 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488139_T1658s_1.ptu
320 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48830_T350s_1.ptu
321 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488175_T2087s_1.ptu
322 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48856_T662s_1.ptu
323 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488230_T2745s_1.ptu
324 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48875_T894s_1.ptu
325 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488127_T1521s_1.ptu
326 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488195_T2327s_1.ptu
327 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48884_T998s_1.ptu
328 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488186_T2219s_1.ptu
329 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48882_T974s_1.ptu
330 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488134_T1606s_1.ptu
331 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488108_T1286s_1.ptu
332 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48827_T315s_1.ptu
333 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488214_T2553s_1.ptu
334 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488163_T1944s_1.ptu
335 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48839_T460s_1.ptu
336 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488186_T2233s_1.ptu
337 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488243_T2902s_1.ptu
338 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488137_T1642s_1.ptu
339 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488156_T1861s_1.ptu
340 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF4882_T14s_1.ptu
341 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488142_T1694s_1.ptu
342 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48811_T123s_1.ptu
343 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488248_T2961s_1.ptu
344 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48895_T1136s_1.ptu
345 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48826_T303s_1.ptu
346 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488191_T2293s_1.ptu
347 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488137_T1634s_1.ptu
348 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488133_T1585s_1.ptu
349 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488216_T2577s_1.ptu
350 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488107_T1274s_1.ptu
351 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488130_T1557s_1.ptu
352 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488120_T1436s_1.ptu
353 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48868_T806s_1.ptu
354 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488125_T1490s_1.ptu
355 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48832_T374s_1.ptu
356 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488119_T1424s_1.ptu
357 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48877_T914s_1.ptu
358 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488188_T2257s_1.ptu
359 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48836_T423s_1.ptu
360 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488176_T2112s_1.ptu
361 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488112_T1334s_1.ptu
362 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48815_T170s_1.ptu
363 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48838_T448s_1.ptu
364 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488178_T2136s_1.ptu
365 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488198_T2363s_1.ptu
366 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48855_T650s_1.ptu
367 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48899_T1178s_1.ptu
368 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488161_T1931s_1.ptu
369 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488246_T2957s_1.ptu
370 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488231_T2757s_1.ptu
371 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488122_T1460s_1.ptu
372 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488157_T1873s_1.ptu
373 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488194_T2329s_1.ptu
374 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48846_T542s_1.ptu
375 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488144_T1718s_1.ptu
376 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488111_T1322s_1.ptu
377 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488143_T1705s_1.ptu
378 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48874_T882s_1.ptu
379 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488124_T1478s_1.ptu
380 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488106_T1268s_1.ptu
381 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48892_T1094s_1.ptu
382 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488115_T1376s_1.ptu
383 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48864_T761s_1.ptu
384 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488239_T2853s_1.ptu
385 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48893_T1106s_1.ptu
386 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488154_T1837s_1.ptu
387 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488105_T1256s_1.ptu
388 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48829_T338s_1.ptu
389 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48871_T845s_1.ptu
390 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48898_T1172s_1.ptu
391 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48880_T954s_1.ptu
392 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48811_T122s_1.ptu
393 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488141_T1682s_1.ptu
394 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488237_T2829s_1.ptu
395 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48810_T110s_1.ptu
396 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48898_T1166s_1.ptu
397 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48812_T135s_1.ptu
398 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48887_T1034s_1.ptu
399 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF488202_T2410s_1.ptu
400 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48828_T326s_1.ptu
401 of 424: /beegfs/ye53nis/data/Pablo_structured_experiment/all_clean_ptu/20 nM AF48886_T1027s_1.ptu
Different binning was chosen for correlation. Loading dataset 1 with bin=100000.0. This can take a while...
Processing correlation of unprocessed dataset 1
Processing correlation with correction by prediction of dataset 1
Loading dataset 2 from path /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu with bin=1e6. This can take a while...
1 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48869_T825s_1.ptu
2 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488196_T2359s_1.ptu
3 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488146_T1755s_1.ptu
4 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488227_T2733s_1.ptu
5 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48823_T287s_1.ptu
6 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48829_T342s_1.ptu
7 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488194_T2334s_1.ptu
8 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48851_T607s_1.ptu
9 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488210_T2528s_1.ptu
10 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488234_T2818s_1.ptu
11 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48882_T982s_1.ptu
12 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488251_T3022s_1.ptu
13 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48848_T610s_1.ptu
14 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488160_T1924s_1.ptu
15 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48846_T584s_1.ptu
16 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488268_T3227s_1.ptu
17 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48823_T268s_1.ptu
18 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48832_T378s_1.ptu
19 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488241_T2902s_1.ptu
20 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488198_T2383s_1.ptu
21 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48869_T823s_1.ptu
22 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48895_T1139s_1.ptu
23 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48878_T934s_1.ptu
24 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488100_T1197s_1.ptu
25 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488155_T1864s_1.ptu
26 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488132_T1585s_1.ptu
27 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48866_T787s_1.ptu
28 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48877_T922s_1.ptu
29 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48861_T728s_1.ptu
30 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488129_T1548s_1.ptu
31 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48825_T292s_1.ptu
32 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488205_T2468s_1.ptu
33 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4889_T106s_1.ptu
34 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48840_T474s_1.ptu
35 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48843_T546s_1.ptu
36 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48887_T1042s_1.ptu
37 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488116_T1391s_1.ptu
38 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4884_T39s_1.ptu
39 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488110_T1318s_1.ptu
40 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488218_T2625s_1.ptu
41 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48870_T834s_1.ptu
42 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488270_T3251s_1.ptu
43 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488177_T2129s_1.ptu
44 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488136_T1633s_1.ptu
45 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48844_T521s_1.ptu
46 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48829_T341s_1.ptu
47 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4887_T75s_1.ptu
48 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48842_T497s_1.ptu
49 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488291_T3505s_1.ptu
50 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488117_T1403s_1.ptu
51 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48814_T159s_1.ptu
52 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488215_T2589s_1.ptu
53 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488128_T1536s_1.ptu
54 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488135_T1621s_1.ptu
55 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48870_T837s_1.ptu
56 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488239_T2878s_1.ptu
57 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48827_T317s_1.ptu
58 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488231_T2781s_1.ptu
59 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488294_T3541s_1.ptu
60 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488247_T2974s_1.ptu
61 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488226_T2721s_1.ptu
62 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488182_T2189s_1.ptu
63 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488248_T2986s_1.ptu
64 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488216_T2601s_1.ptu
65 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48824_T282s_1.ptu
66 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488152_T1827s_1.ptu
67 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48868_T811s_1.ptu
68 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488112_T1342s_1.ptu
69 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48811_T123s_1.ptu
70 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488185_T2225s_1.ptu
71 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48828_T330s_1.ptu
72 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488232_T2793s_1.ptu
73 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488293_T3529s_1.ptu
74 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48825_T294s_1.ptu
75 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4886_T66s_1.ptu
76 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48822_T273s_1.ptu
77 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48889_T1067s_1.ptu
78 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48816_T183s_1.ptu
79 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48879_T946s_1.ptu
80 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48876_T910s_1.ptu
81 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48840_T507s_1.ptu
82 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48849_T583s_1.ptu
83 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48850_T595s_1.ptu
84 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48822_T257s_1.ptu
85 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488184_T2213s_1.ptu
86 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48813_T148s_1.ptu
87 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488121_T1452s_1.ptu
88 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488254_T3059s_1.ptu
89 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48814_T171s_1.ptu
90 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48849_T581s_1.ptu
91 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488122_T1464s_1.ptu
92 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488153_T1839s_1.ptu
93 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48851_T605s_1.ptu
94 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48819_T222s_1.ptu
95 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488104_T1246s_1.ptu
96 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488179_T2153s_1.ptu
97 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488130_T1560s_1.ptu
98 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48874_T885s_1.ptu
99 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488115_T1379s_1.ptu
100 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488282_T3397s_1.ptu
101 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488259_T3119s_1.ptu
102 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48858_T690s_1.ptu
103 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48821_T244s_1.ptu
104 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48825_T313s_1.ptu
105 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48861_T726s_1.ptu
106 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48817_T196s_1.ptu
107 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488265_T3191s_1.ptu
108 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48834_T400s_1.ptu
109 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48854_T643s_1.ptu
110 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48873_T873s_1.ptu
111 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48857_T678s_1.ptu
112 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488208_T2504s_1.ptu
113 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4884_T41s_1.ptu
114 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48810_T120s_1.ptu
115 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48873_T870s_1.ptu
116 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48872_T859s_1.ptu
117 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488180_T2165s_1.ptu
118 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48828_T328s_1.ptu
119 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488224_T2697s_1.ptu
120 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488167_T2009s_1.ptu
121 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488237_T2854s_1.ptu
122 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488258_T3107s_1.ptu
123 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488222_T2673s_1.ptu
124 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48845_T534s_1.ptu
125 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488280_T3373s_1.ptu
126 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48848_T569s_1.ptu
127 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48812_T145s_1.ptu
128 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48844_T522s_1.ptu
129 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48828_T351s_1.ptu
130 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488173_T2081s_1.ptu
131 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48893_T1115s_1.ptu
132 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48865_T774s_1.ptu
133 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48862_T740s_1.ptu
134 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488187_T2250s_1.ptu
135 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48812_T136s_1.ptu
136 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48853_T631s_1.ptu
137 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48843_T509s_1.ptu
138 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48880_T955s_1.ptu
139 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48845_T535s_1.ptu
140 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488133_T1597s_1.ptu
141 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48836_T426s_1.ptu
142 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488138_T1658s_1.ptu
143 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488235_T2830s_1.ptu
144 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488111_T1330s_1.ptu
145 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488156_T1876s_1.ptu
146 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48855_T655s_1.ptu
147 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4887_T80s_1.ptu
148 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48831_T365s_1.ptu
149 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48854_T642s_1.ptu
150 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48872_T861s_1.ptu
151 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488107_T1282s_1.ptu
152 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48824_T280s_1.ptu
153 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488256_T3083s_1.ptu
154 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488204_T2456s_1.ptu
155 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488197_T2371s_1.ptu
156 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48852_T619s_1.ptu
157 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488176_T2117s_1.ptu
158 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488229_T2757s_1.ptu
159 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48842_T498s_1.ptu
160 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48826_T305s_1.ptu
161 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48864_T762s_1.ptu
162 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488289_T3481s_1.ptu
163 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48849_T624s_1.ptu
164 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488249_T2998s_1.ptu
165 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48816_T184s_1.ptu
166 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48817_T197s_1.ptu
167 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48884_T1006s_1.ptu
168 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488178_T2141s_1.ptu
169 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48855_T654s_1.ptu
170 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48813_T158s_1.ptu
171 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488118_T1415s_1.ptu
172 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488127_T1524s_1.ptu
173 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48836_T425s_1.ptu
174 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48830_T352s_1.ptu
175 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48829_T340s_1.ptu
176 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488261_T3143s_1.ptu
177 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48837_T468s_1.ptu
178 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48830_T354s_1.ptu
179 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48819_T220s_1.ptu
180 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488236_T2842s_1.ptu
181 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488257_T3095s_1.ptu
182 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48819_T219s_1.ptu
183 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488253_T3047s_1.ptu
184 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48841_T485s_1.ptu
185 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48830_T377s_1.ptu
186 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488201_T2420s_1.ptu
187 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48835_T412s_1.ptu
188 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488269_T3239s_1.ptu
189 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48846_T545s_1.ptu
190 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48833_T416s_1.ptu
191 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48839_T461s_1.ptu
192 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48850_T637s_1.ptu
193 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488276_T3324s_1.ptu
194 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488286_T3445s_1.ptu
195 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488181_T2177s_1.ptu
196 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488199_T2395s_1.ptu
197 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488244_T2938s_1.ptu
198 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488101_T1209s_1.ptu
199 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48881_T970s_1.ptu
200 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488126_T1512s_1.ptu
201 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488223_T2685s_1.ptu
202 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48847_T557s_1.ptu
203 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488158_T1900s_1.ptu
204 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488139_T1670s_1.ptu
205 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48824_T281s_1.ptu
206 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48859_T702s_1.ptu
207 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48846_T546s_1.ptu
208 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488267_T3215s_1.ptu
209 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4883_T28s_1.ptu
210 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48827_T318s_1.ptu
211 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48820_T232s_1.ptu
212 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48860_T716s_1.ptu
213 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4882_T15s_1.ptu
214 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48896_T1151s_1.ptu
215 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48830_T353s_1.ptu
216 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488262_T3155s_1.ptu
217 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48899_T1187s_1.ptu
218 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488209_T2516s_1.ptu
219 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488150_T1803s_1.ptu
220 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488147_T1767s_1.ptu
221 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488134_T1609s_1.ptu
222 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48815_T171s_1.ptu
223 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488264_T3179s_1.ptu
224 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488298_T3590s_1.ptu
225 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488193_T2322s_1.ptu
226 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4888_T87s_1.ptu
227 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488106_T1270s_1.ptu
228 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488284_T3421s_1.ptu
229 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488145_T1743s_1.ptu
230 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48828_T329s_1.ptu
231 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488183_T2201s_1.ptu
232 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48856_T668s_1.ptu
233 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4888_T93s_1.ptu
234 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4889_T100s_1.ptu
235 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48818_T222s_1.ptu
236 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48835_T442s_1.ptu
237 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488175_T2105s_1.ptu
238 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48841_T486s_1.ptu
239 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48833_T390s_1.ptu
240 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488245_T2950s_1.ptu
241 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488195_T2347s_1.ptu
242 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48867_T799s_1.ptu
243 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48839_T494s_1.ptu
244 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48817_T195s_1.ptu
245 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488105_T1258s_1.ptu
246 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48897_T1163s_1.ptu
247 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48835_T413s_1.ptu
248 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48820_T248s_1.ptu
249 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48841_T520s_1.ptu
250 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4881_T0s_1.ptu
251 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488165_T1984s_1.ptu
252 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48887_T1040s_1.ptu
253 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48817_T210s_1.ptu
254 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4883_T27s_1.ptu
255 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48891_T1091s_1.ptu
256 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488233_T2806s_1.ptu
257 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48811_T132s_1.ptu
258 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48876_T906s_1.ptu
259 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48839_T462s_1.ptu
260 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488109_T1306s_1.ptu
261 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488162_T1948s_1.ptu
262 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48858_T692s_1.ptu
263 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488151_T1815s_1.ptu
264 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488203_T2444s_1.ptu
265 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48815_T184s_1.ptu
266 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48831_T366s_1.ptu
267 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48883_T991s_1.ptu
268 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48881_T967s_1.ptu
269 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488287_T3457s_1.ptu
270 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48884_T1003s_1.ptu
271 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48886_T1031s_1.ptu
272 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48820_T234s_1.ptu
273 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48845_T533s_1.ptu
274 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488100_T1199s_1.ptu
275 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488214_T2577s_1.ptu
276 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48845_T571s_1.ptu
277 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48856_T666s_1.ptu
278 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488172_T2069s_1.ptu
279 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48837_T438s_1.ptu
280 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48822_T256s_1.ptu
281 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488243_T2926s_1.ptu
282 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488240_T2890s_1.ptu
283 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488206_T2480s_1.ptu
284 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48818_T208s_1.ptu
285 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48866_T788s_1.ptu
286 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48863_T752s_1.ptu
287 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488238_T2866s_1.ptu
288 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488260_T3131s_1.ptu
289 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48818_T210s_1.ptu
290 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48892_T1103s_1.ptu
291 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48813_T147s_1.ptu
292 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48834_T402s_1.ptu
293 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48853_T630s_1.ptu
294 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488114_T1367s_1.ptu
295 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488300_T3614s_1.ptu
296 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488119_T1427s_1.ptu
297 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48868_T813s_1.ptu
298 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488157_T1888s_1.ptu
299 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488188_T2262s_1.ptu
300 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488212_T2552s_1.ptu
301 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48833_T388s_1.ptu
302 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48864_T764s_1.ptu
303 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48823_T269s_1.ptu
304 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488252_T3035s_1.ptu
305 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488169_T2032s_1.ptu
306 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48890_T1079s_1.ptu
307 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48871_T847s_1.ptu
308 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488137_T1646s_1.ptu
309 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488166_T1997s_1.ptu
310 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488283_T3409s_1.ptu
311 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488279_T3360s_1.ptu
312 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488125_T1500s_1.ptu
313 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488200_T2408s_1.ptu
314 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48882_T979s_1.ptu
315 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488281_T3385s_1.ptu
316 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48811_T124s_1.ptu
317 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488102_T1221s_1.ptu
318 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48825_T293s_1.ptu
319 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488186_T2237s_1.ptu
320 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488274_T3299s_1.ptu
321 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48838_T450s_1.ptu
322 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48826_T304s_1.ptu
323 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488163_T1960s_1.ptu
324 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488174_T2093s_1.ptu
325 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488250_T3010s_1.ptu
326 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488255_T3071s_1.ptu
327 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48847_T597s_1.ptu
328 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488288_T3469s_1.ptu
329 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488103_T1234s_1.ptu
330 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488164_T1972s_1.ptu
331 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48885_T1018s_1.ptu
332 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48821_T246s_1.ptu
333 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48812_T135s_1.ptu
334 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488242_T2914s_1.ptu
335 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488142_T1706s_1.ptu
336 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488159_T1912s_1.ptu
337 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4887_T76s_1.ptu
338 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48886_T1028s_1.ptu
339 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488263_T3167s_1.ptu
340 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48824_T299s_1.ptu
341 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488113_T1354s_1.ptu
342 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488108_T1294s_1.ptu
343 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48816_T197s_1.ptu
344 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48863_T750s_1.ptu
345 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488140_T1682s_1.ptu
346 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488230_T2769s_1.ptu
347 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488131_T1573s_1.ptu
348 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488220_T2649s_1.ptu
349 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48838_T449s_1.ptu
350 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488154_T1852s_1.ptu
351 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48838_T481s_1.ptu
352 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488120_T1439s_1.ptu
353 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488207_T2492s_1.ptu
354 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488171_T2056s_1.ptu
355 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488190_T2286s_1.ptu
356 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48847_T558s_1.ptu
357 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48898_T1175s_1.ptu
358 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48862_T738s_1.ptu
359 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48816_T186s_1.ptu
360 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488299_T3602s_1.ptu
361 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488219_T2637s_1.ptu
362 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48810_T111s_1.ptu
363 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488278_T3348s_1.ptu
364 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488189_T2274s_1.ptu
365 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48859_T704s_1.ptu
366 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48832_T376s_1.ptu
367 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48865_T776s_1.ptu
368 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488225_T2709s_1.ptu
369 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48871_T849s_1.ptu
370 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48883_T994s_1.ptu
371 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48831_T390s_1.ptu
372 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488272_T3276s_1.ptu
373 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488148_T1779s_1.ptu
374 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48875_T898s_1.ptu
375 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48842_T532s_1.ptu
376 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488290_T3493s_1.ptu
377 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48877_T918s_1.ptu
378 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488297_T3578s_1.ptu
379 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488191_T2298s_1.ptu
380 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48814_T160s_1.ptu
381 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48844_T558s_1.ptu
382 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488221_T2661s_1.ptu
383 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488211_T2540s_1.ptu
384 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48821_T260s_1.ptu
385 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488123_T1475s_1.ptu
386 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48852_T617s_1.ptu
387 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488149_T1791s_1.ptu
388 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488141_T1694s_1.ptu
389 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48827_T338s_1.ptu
390 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488217_T2613s_1.ptu
391 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48848_T571s_1.ptu
392 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4886_T63s_1.ptu
393 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48843_T510s_1.ptu
394 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48810_T112s_1.ptu
395 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488124_T1488s_1.ptu
396 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF488271_T3263s_1.ptu
397 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48840_T473s_1.ptu
398 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48826_T306s_1.ptu
399 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF4885_T54s_1.ptu
400 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48827_T316s_1.ptu
401 of 440: /beegfs/ye53nis/data/Pablo_structured_experiment/all_dirty_ptu/DiO LUV 10uM in 20 nM AF48850_T593s_1.ptu
Different binning was chosen for correlation. Loading dataset 2 with bin=100000.0. This can take a while...
Processing correlation of unprocessed dataset 2
Processing correlation with correction by prediction of dataset 2
/beegfs/ye53nis/drmed-git/src/fluotracify/applications/correction.py:508: UserWarning: Metadata is not saved with data. Reason: the correlation algorithm failed for one or more traces which were shorter than 32 time steps after correction.Since metadata is loaded in the beginning, it is not sure, which correlation is missing to ensure proper joining of data and metadata.
  warnings.warn('Metadata is not saved with data. Reason: the '
corr_out = pd.read_csv(filepath_or_buffer='data/exp-210807-hparams/2021-08-26_correlations.csv')
corr_out
  \(D\) in \(\frac{{\mu m^2}}{{s}}\) \(\tau_{{D}}\) in \(ms\) Trace lengths folderid-tracesused Photon count bin for correlation in \(ns\)
0 19.078159 0.590783 8192.0 0-orig 100000.0
1 20.314897 0.554817 8192.0 0-orig 100000.0
2 21.007772 0.536518 8192.0 0-orig 100000.0
3 22.716317 0.496166 8192.0 0-orig 100000.0
4 24.089236 0.467888 8192.0 0-orig 100000.0
4795 0.577566 19.514749 7240.0 1-pred-0.9 100000.0
4796 0.236147 47.728941 7078.0 1-pred-0.9 100000.0
4797 1.860295 6.058745 7819.0 1-pred-0.9 100000.0
4798 1.946114 5.791570 7520.0 1-pred-0.9 100000.0
4799 0.859799 13.108948 7841.0 1-pred-0.9 100000.0

4800 rows × 5 columns

  1. Check out the nan values
    corr_out[corr_out['$D$ in $\\frac{{\mu m^2}}{{s}}$'].isna()]
    

    \(D\) in \(\frac{{\mu m^2}}{{s}}\) \(\tau_{{D}}\) in \(ms\) Trace lengths folderid-tracesused Photon count bin for correlation in \(ns\)

    Good news! This time there are no nan values. In exp-210204-unet, we had 31 of them in the clean dataset with pred-0.1, which suggested a lot of false positives in the clean data, which led to traces shorter than 32 time steps, where the multipletau algorithm fails. Still, I am not sure why the metadata was not saved out in the correlation program. I will investigate another time…

  2. Refactor folder-id_traces-used into 2 columns
    corr_out['folder_id-traces_used']
    
    0           0-orig
    1           0-orig
    2           0-orig
    3           0-orig
    4           0-orig
               ...
    4795    1-pred-0.9
    4796    1-pred-0.9
    4797    1-pred-0.9
    4798    1-pred-0.9
    4799    1-pred-0.9
    Name: folder_id-traces_used, Length: 4800, dtype: object
    
    corr_out[['Folder ID', 'Traces used']] = corr_out['folder_id-traces_used'].str.split(pat='-', n=1, expand=True)
    
  3. Some basic statistical descriptions
    corr_stats = pd.DataFrame()
    for fid in sorted(set(corr_out['Folder ID'])):
        for tu in sorted(set(corr_out['Traces used'])):
            corr_new = corr_out[(corr_out['Folder ID'] == fid) & (corr_out['Traces used'] == tu)].describe()
            new_index_tuple = zip(((fid),)*8, ((tu),)*8, ('count', 'mean', 'std', 'min', '25%', '50%', '75%', 'max'))
            corr_new.index = pd.MultiIndex.from_tuples(new_index_tuple, names=['Folder ID', 'Traces used', 'Stats'])
            corr_stats = pd.concat([corr_stats, corr_new], axis=0)
    with pd.option_context('display.max_rows', None, 'display.max_columns', None):
        display(corr_stats)
    
    $D$ in $\frac{{\mu m^2}}{{s}}$ $\tau_{{D}}$ in $ms$ Trace lengths Photon count bin for correlation in $ns$
    Folder ID Traces used Stats
    0 orig count 400.000000 400.000000 400.000000 400.0
    mean 22.397754 0.507013 8192.000000 100000.0
    std 1.994998 0.042972 0.000000 0.0
    min 18.207878 0.379407 8192.000000 100000.0
    25% 20.921029 0.481523 8192.000000 100000.0
    50% 22.149393 0.508866 8192.000000 100000.0
    75% 23.407135 0.538743 8192.000000 100000.0
    max 29.707022 0.619021 8192.000000 100000.0
    pred-0.1 count 400.000000 400.000000 400.000000 400.0
    mean 22.444596 0.502583 8182.360000 100000.0
    std 0.645526 0.014330 8.417990 0.0
    min 20.681787 0.460807 8124.000000 100000.0
    25% 21.961521 0.492386 8179.000000 100000.0
    50% 22.368956 0.503870 8184.000000 100000.0
    75% 22.890670 0.513218 8188.000000 100000.0
    max 24.459398 0.544975 8192.000000 100000.0
    pred-0.3 count 400.000000 400.000000 400.000000 400.0
    mean 22.420126 0.503131 8190.077500 100000.0
    std 0.644112 0.014326 2.892648 0.0
    min 20.668056 0.461330 8174.000000 100000.0
    25% 21.946382 0.493151 8189.000000 100000.0
    50% 22.342680 0.504463 8191.000000 100000.0
    75% 22.855193 0.513572 8192.000000 100000.0
    max 24.431668 0.545337 8192.000000 100000.0
    pred-0.5 count 400.000000 400.000000 400.000000 400.0
    mean 22.411363 0.503324 8191.300000 100000.0
    std 0.641300 0.014276 1.488469 0.0
    min 20.666124 0.462115 8183.000000 100000.0
    25% 21.944307 0.493346 8191.000000 100000.0
    50% 22.325232 0.504857 8192.000000 100000.0
    75% 22.846137 0.513621 8192.000000 100000.0
    max 24.390149 0.545388 8192.000000 100000.0
    pred-0.7 count 400.000000 400.000000 400.000000 400.0
    mean 22.403669 0.503500 8191.810000 100000.0
    std 0.643185 0.014327 0.797175 0.0
    min 20.615435 0.462115 8184.000000 100000.0
    25% 21.935997 0.493728 8192.000000 100000.0
    50% 22.323219 0.504903 8192.000000 100000.0
    75% 22.828459 0.513816 8192.000000 100000.0
    max 24.390149 0.546729 8192.000000 100000.0
    pred-0.9 count 400.000000 400.000000 400.000000 400.0
    mean 22.401607 0.503547 8191.937500 100000.0
    std 0.643493 0.014338 0.509158 0.0
    min 20.615435 0.462115 8185.000000 100000.0
    25% 21.917571 0.493728 8192.000000 100000.0
    50% 22.323219 0.504903 8192.000000 100000.0
    75% 22.828459 0.514247 8192.000000 100000.0
    max 24.390149 0.546729 8192.000000 100000.0
    1 orig count 400.000000 400.000000 400.000000 400.0
    mean 2.089838 106.604322 8192.000000 100000.0
    std 4.867448 244.689593 0.000000 0.0
    min 0.005636 0.370469 8192.000000 100000.0
    25% 0.110612 9.232342 8192.000000 100000.0
    50% 0.277676 40.594163 8192.000000 100000.0
    75% 1.220995 101.904123 8192.000000 100000.0
    max 30.423708 2000.000000 8192.000000 100000.0
    pred-0.1 count 400.000000 400.000000 400.000000 400.0
    mean 23.487424 0.729772 6014.930000 100000.0
    std 11.632062 0.978357 655.329984 0.0
    min 0.704858 0.213925 4055.000000 100000.0
    25% 14.296784 0.349318 5582.750000 100000.0
    50% 22.055818 0.511024 6069.500000 100000.0
    75% 32.266393 0.788363 6477.250000 100000.0
    max 52.686940 15.990532 7586.000000 100000.0
    pred-0.3 count 400.000000 400.000000 400.000000 400.0
    mean 16.200632 1.809016 6649.735000 100000.0
    std 10.841992 4.819835 535.204704 0.0
    min 0.204291 0.221455 4702.000000 100000.0
    25% 7.332725 0.495671 6317.250000 100000.0
    50% 14.302030 0.788074 6699.500000 100000.0
    75% 22.739144 1.537091 7043.250000 100000.0
    max 50.895517 55.171591 7781.000000 100000.0
    pred-0.5 count 400.000000 400.000000 400.000000 400.0
    mean 10.787920 4.217836 7002.527500 100000.0
    std 8.761952 11.682295 448.347782 0.0
    min 0.080967 0.273684 5168.000000 100000.0
    25% 3.713476 0.730534 6733.500000 100000.0
    50% 8.729210 1.291190 7049.000000 100000.0
    75% 15.428534 3.035206 7327.750000 100000.0
    max 41.182749 139.205234 7887.000000 100000.0
    pred-0.7 count 400.000000 400.000000 400.000000 400.0
    mean 6.669685 9.416059 7267.402500 100000.0
    std 6.356652 21.185204 368.356589 0.0
    min 0.065038 0.375238 5715.000000 100000.0
    25% 1.657642 1.154375 7053.500000 100000.0
    50% 4.786510 2.355174 7323.000000 100000.0
    75% 9.763811 6.803933 7543.250000 100000.0
    max 30.037104 173.299931 7969.000000 100000.0
    pred-0.9 count 400.000000 400.000000 400.000000 400.0
    mean 3.074691 21.394545 7556.762500 100000.0
    std 3.639749 36.676351 269.188535 0.0
    min 0.038601 0.593323 6349.000000 100000.0
    25% 0.459359 2.485594 7406.750000 100000.0
    50% 1.544961 7.295842 7603.500000 100000.0
    75% 4.534553 24.536530 7750.000000 100000.0
    max 18.996487 291.988834 8072.000000 100000.0
  4. Have a look at the trace lengths
    x = 'Trace lengths'
    
    g = sns.FacetGrid(data=corr_out,
                      col='Folder ID',
                      col_wrap=1,
                      sharex=True,
                      aspect=1.5,
                      height=5,
                      legend_out=True)
    g.map_dataframe(sns.boxplot,
          x=x,
          y='Traces used',
          palette='colorblind',
          showfliers=False)
    g.map_dataframe(sns.stripplot,
          x=x,
          y='Traces used',
          dodge=True,
          palette=sns.color_palette(['0.3']),
          size=4,
          jitter=0.2)
    g.set_xlabels(x)
    g.fig.patch.set_facecolor('white')
    g.tight_layout()
    plt.show()
    

    Trace lengths: 2021-08-26_correlations_tracelength.png

    First of all: the extension of simulated training data clearly helped to avoid the enormous amount of false positives in the run exp-210204-unet in folder 0. Here, only few traces are shortened, which is to be expected, since it represents a negative control. Looking at folder 1, the reduction of trace length increased with all prediction thresholds. I later plot trace length vs tau to see if this reduction in trace length introduces artifacts or if it is mainly a desired outcome of more found artifacts.

  5. Have a look at diffusion coefficient and transit times
    # '$D$ in $\\frac{{\mu m^2}}{{s}}$'
    # '$\\tau_{{D}}$ in $ms$'
    x = '$D$ in $\\frac{{\mu m^2}}{{s}}$'
    
    g = sns.FacetGrid(data=corr_out,
                      col='Folder ID',
                      col_wrap=1,
                      sharex=True,
                      aspect=1.5,
                      height=5,
                      legend_out=True)
    g.map_dataframe(sns.boxplot,
          x=x,
          y='Traces used',
          palette='colorblind',
          showfliers=False).set(xscale = 'log')
    g.map_dataframe(sns.stripplot,
          x=x,
          y='Traces used',
          dodge=True,
          palette=sns.color_palette(['0.3']),
          size=4,
          jitter=0.2).set(xscale = 'log')
    g.set_xlabels(x)
    g.tight_layout()
    g.fig.patch.set_facecolor('white')
    plt.show()
    

    tau: 2021-08-26_correlations_tau.png D: 2021-08-26_correlations_diffrate.png

  6. This scatterplot shows Diffusion rates / transit times against trace lengths. I used a subsample to avoid overplotting.
    # '$D$ in $\\frac{{\mu m^2}}{{s}}$'
    # '$\\tau_{{D}}$ in $ms$'
    x = '$\\tau_{{D}}$ in $ms$'
    
    g = sns.FacetGrid(data=corr_out,
                      row='Traces used',
                      col='Folder ID',
                      hue='Traces used',
                      sharex=True,
                      sharey=True,
                      aspect=1.5,
                      height=3.5,
                      margin_titles=True,
                      legend_out=True)
    g.map_dataframe(sns.scatterplot,
          x=x,
          y='Trace lengths',
          palette='colorblind').set(xscale = 'log')
    g.add_legend(title='Traces used')
    g.set_xlabels(x)
    g.tight_layout()
    g.fig.patch.set_facecolor('white')
    plt.show()
    

    2021-08-26_correlations_tlvstau.png

    # '$D$ in $\\frac{{\mu m^2}}{{s}}$'
    # '$\\tau_{{D}}$ in $ms$'
    x = '$\\tau_{{D}}$ in $ms$'
    
    g = sns.FacetGrid(data=corr_out.sample(1000),
                      row='Folder ID',
                      hue='Traces used',
                      hue_order=['orig', 'pred-0.1', 'pred-0.3', 'pred-0.5', 'pred-0.7', 'pred-0.9'],
                      sharex=True,
                      sharey=True,
                      aspect=1.5,
                      height=4,
                      margin_titles=True,
                      legend_out=True)
    g.map_dataframe(sns.scatterplot,
          x=x,
          y='Trace lengths',
          palette='colorblind').set(xscale = 'log')
    g.add_legend(title='Traces used')
    g.set_xlabels(x)
    g.tight_layout()
    g.fig.patch.set_facecolor('white')
    plt.show()
    

    2021-08-26_correlations_tlvstau2.png

    For me, this plot shows that the reduction of trace length is not too severe even with low prediction thresholds. Also, the reduction in trace length mainly seems to be due to better recognition of artifacts.

2.4.7 Apply model from run 2 to simulated data

%cd /beegfs/ye53nis/drmed-git
/beegfs/ye53nis/drmed-git
from pathlib import Path

import sys
import mlflow
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import tensorflow as tf
print("tf version: ", tf.version.VERSION)
print("tf.keras version: ", tf.keras.__version__)
print("mlflow version: ", mlflow.version.VERSION)
tf version:  2.5.0
tf.keras version:  2.5.0
mlflow version:  1.19.0
sys.path.append('src/')
from fluotracify.simulations import (
   import_simulation_from_csv as isfc,
   analyze_simulations as ans,
)
from fluotracify.training import build_model as bm, preprocess_data as ppd
from fluotracify.applications import correlate, plots, correction
from fluotracify.imports import ptu_utils as ptu
folder = '/beegfs/ye53nis/saves/firstartifact_Nov2020_test'
col_per_example = 3
lab_thresh = 0.04
pred_thresh = 0.5
artifact = 0
model_type = 1
fwhm = 250
logged_model = 'file:///beegfs/ye53nis/drmed-git/data/mlruns/8/00f2635d9fa2463c9a066722163405be/artifacts/model'
metrics_thresholds = [0.1, 0.3, 0.5, 0.7, 0.9]
loaded_model = mlflow.keras.load_model(logged_model, compile=False)
loaded_model.compile(loss=bm.binary_ce_dice_loss(),
                     optimizer=tf.keras.optimizers.Adam(),
                     metrics = bm.unet_metrics(metrics_thresholds))
loaded_model
WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
<tensorflow.python.keras.engine.functional.Functional at 0x2b01891b62e0>
# The following block has to be executed multiple times with different array lengths
# e.g. 2**14, 2**13, 2**12
# note that: after executing 2**14, then 2**13, there will be an error
# if you then execute 2**12 and then 2**13 again, the error will be gone
test_features = np.zeros((2**13)).reshape(1, -1, 1)
print(test_features.shape)
predictions = loaded_model.predict(test_features, verbose=0).flatten()
predictions
(1, 8192, 1)
array([1.4064237e-01, 1.3650321e-08, 7.4799689e-10, ..., 3.1755112e-10,
       1.8236226e-10, 6.6588524e-10], dtype=float32)

The simulated data was separated in train, val, and test beforehand. Now, I only load test data.

dataset, _, nsamples, experiment_params = isfc.import_from_csv(
    folder=folder,
    header=12,
    frac_train=1,
    col_per_example=col_per_example,
    dropindex=None,
    dropcolumns=None)
0 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.069/1.0/traces_brightclust_Nov2020_D0.069_set010.csv
1 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/50/0.1/traces_brightclust_Nov2020_D50_set003.csv
2 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.4/0.1/traces_brightclust_Nov2020_D0.4_set001.csv
3 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.2/0.1/traces_brightclust_Nov2020_D0.2_set007.csv
4 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/3.0/1.0/traces_brightclust_Nov2020_D3.0_set002.csv
5 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/3.0/0.01/traces_brightclust_Nov2020_D3.0_set004.csv
6 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/50/0.01/traces_brightclust_Nov2020_D50_set002.csv
7 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.2/1.0/traces_brightclust_Nov2020_D0.2_set005.csv
8 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.6/1.0/traces_brightclust_Nov2020_D0.6_set009.csv
9 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/10/0.1/traces_brightclust_Nov2020_D10_set001.csv10 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.08/1.0/traces_brightclust_Nov2020_D0.08_set001.csv
11 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.6/0.1/traces_brightclust_Nov2020_D0.6_set003.csv
12 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.1/1.0/traces_brightclust_Nov2020_D0.1_set001.csv
13 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.4/1.0/traces_brightclust_Nov2020_D0.4_set005.csv
14 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/10/1.0/traces_brightclust_Nov2020_D10_set005.csv
15 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/1.0/1.0/traces_brightclust_Nov2020_D1.0_set005.csv
16 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.069/0.1/traces_brightclust_Nov2020_D0.069_set001.csv
17 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/50/1.0/traces_brightclust_Nov2020_D50_set001.csv
18 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.1/0.01/traces_brightclust_Nov2020_D0.1_set002.csv
19 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.08/0.1/traces_brightclust_Nov2020_D0.08_set003.csv
20 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/1.0/0.01/traces_brightclust_Nov2020_D1.0_set006.csv
21 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/1.0/0.1/traces_brightclust_Nov2020_D1.0_set003.csv
22 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.2/0.01/traces_brightclust_Nov2020_D0.2_set002.csv
23 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.1/0.1/traces_brightclust_Nov2020_D0.1_set005.csv
24 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/3.0/0.1/traces_brightclust_Nov2020_D3.0_set007.csv
25 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.08/0.01/traces_brightclust_Nov2020_D0.08_set005.csv
26 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.069/0.01/traces_brightclust_Nov2020_D0.069_set005.csv
27 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/10/0.01/traces_brightclust_Nov2020_D10_set002.csv
28 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.6/0.01/traces_brightclust_Nov2020_D0.6_set008.csv
29 /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.4/0.01/traces_brightclust_Nov2020_D0.4_set008.csv
experiment_params.T.sort_values(by='diffusion rate of molecules in micrometer^2 / s', ignore_index=True)
  unique identifier path and file name FWHMs of excitation PSFs used in nm Extent of simulated PSF (distance to center of Gaussian) in nm total simulation time in ms time step in ms number of fast molecules diffusion rate of molecules in micrometer2 / s width of the simulation in nm height of the simulation in nm number of slow clusters diffusion rate of clusters in micrometer2 / s trace001
0 bebf4d6b-0ffe-4dcf-be78-93bfa70dd04c /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 679 0.069 3000.0 3000.0 3 1.0 label0011
1 5abeceef-c30b-46d8-96a6-dfd623820542 /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 1930 0.069 3000.0 3000.0 10 0.01 label0011
2 daf0898c-38d3-48e5-8d97-41490fb30cd7 /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 897 0.069 3000.0 3000.0 7 0.1 label0011
3 cae1a0a7-6d83-49fe-b0b7-fb1f7fa3d541 /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 1673 0.08 3000.0 3000.0 10 0.01 label0011
4 b85aeec3-5e79-444f-b0c5-5ba7057fad8c /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 856 0.08 3000.0 3000.0 7 0.1 label0011
5 9f7d64ef-5e20-4a2a-bf66-adefc770d2d3 /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 2569 0.08 3000.0 3000.0 3 1.0 label0011
6 0d5dcb67-7e31-43dd-8036-11b7243daf14 /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 1156 0.1 3000.0 3000.0 7 0.1 label0011
7 9a2f89bd-c64e-4324-910c-46a5b8a657ba /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 2975 0.1 3000.0 3000.0 10 0.01 label0011
8 db48ccc0-e770-4570-8cf0-d0ea83bfc588 /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 1456 0.1 3000.0 3000.0 3 1.0 label0011
9 33ee5002-7064-4882-88de-79c773d741cd /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 895 0.2 3000.0 3000.0 7 0.1 label0011
10 e3fd87db-7a9e-4f0a-a7f3-30acd95752eb /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 1844 0.2 3000.0 3000.0 10 0.01 label0011
11 511e911a-aaa3-4d9d-a14e-d172bd4d8dbb /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 2735 0.2 3000.0 3000.0 3 1.0 label0011
12 eb3f3035-955f-4e86-bfbd-16421eede63f /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 560 0.4 3000.0 3000.0 3 1.0 label0011
13 20f0a9df-2c4d-409f-86de-ceda40d1aafe /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 1724 0.4 3000.0 3000.0 10 0.01 label0011
14 2c825894-5e96-4acb-9d4d-c6d0c21e71bf /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 1939 0.4 3000.0 3000.0 7 0.1 label0011
15 dd3aef12-414f-4cb7-9330-8db192d67143 /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 2545 0.6 3000.0 3000.0 10 0.01 label0011
16 87d9374a-3ae4-4dfb-8dc3-ec401df4961d /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 1296 0.6 3000.0 3000.0 3 1.0 label0011
17 f43754f2-e756-405f-89a4-9eae43dbf996 /beegfs/ye53nis/saves/firstartifactNov2020/0…. [250] 4000 16384 1.0 2535 0.6 3000.0 3000.0 7 0.1 label0011
18 84116b56-7264-4f63-833a-2fa1af968288 /beegfs/ye53nis/saves/firstartifactNov2020/1…. [250] 4000 16384 1.0 2059 1.0 3000.0 3000.0 3 1.0 label0011
19 c1a85eb4-f202-40f7-8bce-cd3fd3a843dc /beegfs/ye53nis/saves/firstartifactNov2020/1…. [250] 4000 16384 1.0 1357 1.0 3000.0 3000.0 10 0.01 label0011
20 e0118d21-73cf-4c2d-bc3f-f48a44e1b5b8 /beegfs/ye53nis/saves/firstartifactNov2020/1…. [250] 4000 16384 1.0 627 1.0 3000.0 3000.0 7 0.1 label0011
21 6a3c00e3-24e6-4758-9c6d-f2a9dc23f7d9 /beegfs/ye53nis/saves/firstartifactNov2020/10… [250] 4000 16384 1.0 1376 10 3000.0 3000.0 7 0.1 label0011
22 9246b720-b8d4-4390-82b6-71404ced9bea /beegfs/ye53nis/saves/firstartifactNov2020/10… [250] 4000 16384 1.0 1396 10 3000.0 3000.0 10 0.01 label0011
23 e26d94e4-c2d0-46da-b571-91b8fba15f0a /beegfs/ye53nis/saves/firstartifactNov2020/10… [250] 4000 16384 1.0 2618 10 3000.0 3000.0 3 1.0 label0011
24 cf57c545-933f-41b8-bcfe-19723d79b7bf /beegfs/ye53nis/saves/firstartifactNov2020/3…. [250] 4000 16384 1.0 1785 3.0 3000.0 3000.0 10 0.01 label0011
25 f2dcf7ee-f180-4b26-9706-526ceb6b9fe6 /beegfs/ye53nis/saves/firstartifactNov2020/3…. [250] 4000 16384 1.0 2900 3.0 3000.0 3000.0 3 1.0 label0011
26 4f56e86f-7912-4308-b273-1d2cbe919555 /beegfs/ye53nis/saves/firstartifactNov2020/3…. [250] 4000 16384 1.0 3042 3.0 3000.0 3000.0 7 0.1 label0011
27 4a6c04ea-6d95-4c9a-8985-4e2845cd4372 /beegfs/ye53nis/saves/firstartifactNov2020/50… [250] 4000 16384 1.0 3083 50 3000.0 3000.0 3 1.0 label0011
28 0953132e-d7d6-44bb-b1ee-df54b288beb0 /beegfs/ye53nis/saves/firstartifactNov2020/50… [250] 4000 16384 1.0 2323 50 3000.0 3000.0 10 0.01 label0011
29 b2b78a0f-278b-4b7d-82a3-5e625fbc91de /beegfs/ye53nis/saves/firstartifactNov2020/50… [250] 4000 16384 1.0 2561 50 3000.0 3000.0 7 0.1 label0011
diffrates = experiment_params.loc['diffusion rate of molecules in micrometer^2 / s'].astype(np.float32)
nmols = experiment_params.loc['number of fast molecules'].astype(np.float32)
clusters = experiment_params.loc['diffusion rate of clusters in micrometer^2 / s'].astype(np.float32)

dataset_sep = isfc.separate_data_and_labels(array=dataset,
                                            nsamples=nsamples,
                                            col_per_example=col_per_example)

features = dataset_sep['0']
labels_artifact = dataset_sep['1']
labels_artifact_bool = labels_artifact > lab_thresh
labels_puretrace = dataset_sep['2']
The given DataFrame was split into 3 parts with shapes: [(16384, 3000), (16384, 3000), (16384, 3000)]

Let's correct the traces with the new model and correlate them!

  corr_out = ans.correlate_simulations_corrected_by_prediction(
      model=loaded_model,
      lab_thresh=lab_thresh,
      pred_thresh=pred_thresh,
      artifact=artifact,
      model_type=model_type,
      experiment_params=experiment_params,
      nsamples=nsamples,
      features=features,
      labels_artifact=labels_artifact,
      labels_puretrace=labels_puretrace,
      save_as_csv=True)
corr_out
processed correlation of 3000 traces with correction by label
processed correlation of 3000 traces with correction by prediction
processed correlation of 3000 traces without correction
processed correlation of pure 3000 traces
corr_out = pd.read_csv(filepath_or_buffer='data/exp-210807-hparams/2021-12-15_correlations.csv')
corr_out
  Simulated \(D\) Simulated \(D_{{clust}}\) nmol \(D\) in \(\frac{{\mu m^2}}{{s}}\) \(\tau_{{D}}\) in \(ms\) Trace lengths Traces used
0 0.069 1.00 679.0 1.037836 10.860146 16384 corrupted without correction
1 0.069 1.00 679.0 0.814030 13.846002 16384 corrupted without correction
2 0.069 1.00 679.0 1.053716 10.696482 16384 corrupted without correction
3 0.069 1.00 679.0 0.897022 12.564974 16384 corrupted without correction
4 0.069 1.00 679.0 1.083577 10.401707 16384 corrupted without correction
11995 0.400 0.01 1724.0 0.337877 33.358455 16267 corrected by prediction
11996 0.400 0.01 1724.0 0.317319 35.519585 7942 corrected by prediction
11997 0.400 0.01 1724.0 0.380969 29.585216 5315 corrected by prediction
11998 0.400 0.01 1724.0 0.376623 29.926600 12591 corrected by prediction
11999 0.400 0.01 1724.0 0.167653 67.228585 13305 corrected by prediction

12000 rows × 7 columns

  1. Check out the NaN values.
    print('All NaN values: {}'.format(corr_out[corr_out['$D$ in $\\frac{{\mu m^2}}{{s}}$'].isna()]))
    
    All NaN values: Empty DataFrame
    Columns: [Simulated $D$, Simulated $D_{{clust}}$, nmol, $D$ in $\frac{{\mu m^2}}{{s}}$, $\tau_{{D}}$ in $ms$, Trace lengths, Traces used]
    Index: []
    

    We confirmed that the correlation did not fail on none of the test traces (no NaN values).

  2. Now, first plot the trace lengths.
    x = 'Trace lengths'
    
    g = sns.FacetGrid(data=corr_out,
                      col='Simulated $D$',
                      col_wrap=2,
                      sharex=True,
                      aspect=1.5,
                      height=5,
                      legend_out=True)
    g.map_dataframe(sns.boxplot,
          x=x,
          y='Traces used',
          hue='Simulated $D_{{clust}}$',
          palette='colorblind',
          showfliers=False)
    g.add_legend(title='Simulated $D_{{clust}}$')
    g.map_dataframe(sns.stripplot,
          x=x,
          y='Traces used',
          hue='Simulated $D_{{clust}}$',
          dodge=True,
          palette=sns.color_palette(['0.3']),
          size=4,
          jitter=0.2)
    g.set_xlabels(x)
    g.fig.patch.set_facecolor('white')
    g.tight_layout()
    plt.show()
    

    Check out the trace lengths. The reduction seems similar over all simulated data. This speaks to a good generalization. Compared to exp-210204-unet, there is especially an improvement in the Simulated D of 0.1, and 0.4 to 50 and Simulated \(D_{clust}\) of 0.01. 2021-12-15_correlations_trace-lengths.png

  3. Now let's take a look at the diffrates and transit times. Since the distribution follows a log normal, use a log scale.
    # '$D$ in $\\frac{{\mu m^2}}{{s}}$'
    # '$\\tau_{{D}}$ in $ms$'
    x = '$\\tau_{{D}}$ in $ms$'
    
    g = sns.FacetGrid(data=corr_out,
                      col='Simulated $D$',
                      col_wrap=2,
                      sharex=False,  # False for x=D or x=tau, True for x=Trace lengths
                      aspect=1.5,
                      height=5,
                      legend_out=True)
    g.map_dataframe(sns.boxplot,
          x=x,
          y='Traces used',
          hue='Simulated $D_{{clust}}$',
          palette='colorblind',
          showfliers=False).set(xscale = 'log')
    g.add_legend(title='Simulated $D_{{clust}}$')
    g.map_dataframe(sns.stripplot,
          x=x,
          y='Traces used',
          hue='Simulated $D_{{clust}}$',
          dodge=True,
          palette=sns.color_palette(['0.3']),
          size=4,
          jitter=0.2).set(xscale = 'log')
    g.set_xlabels(x)
    g.fig.patch.set_facecolor('white')
    g.tight_layout()
    plt.show()
    

    D: 2021-12-15_correlations_diffrates.png

    We see again a strong improvement in \(D_{clust} = 0.01\), this time with all simulated \(D\). We also see that there are occasional outliers in the prediction group (e.g. \(D = 50\) and \(D_{clust} = 0.01\)), but the statistics are in general okay now.

    Still things to look out for: for \(D=50\) we see a slight distortion to lower transit times / faster diffrates, in both reduction by label and prediction. Here seems to lie a boundary of the correction method.

  4. Lastly, let's take a look at a scatterplot of transit times vs trace lengths using a subsample to avoid overplotting.
    corr_scatter = corr_out[corr_out['Traces used'].isin(['corrected by labels (control)', 'corrected by prediction'])].sample(1000)
    corr_scatter
    
      Simulated \(D\) Simulated \(D_{{clust}}\) nmol \(D\) in \(\frac{{\mu m^2}}{{s}}\) \(\tau_{{D}}\) in \(ms\) Trace lengths Traces used
    9099 0.069 1.00 679.0 0.053497 210.684366 15232 corrected by prediction
    9032 0.069 1.00 679.0 0.065549 171.949847 15583 corrected by prediction
    10678 0.069 0.10 897.0 0.023812 473.327986 15480 corrected by prediction
    9556 3.000 0.01 1785.0 3.321410 3.393455 13251 corrected by prediction
    11319 0.100 0.10 1156.0 0.075619 149.050995 14235 corrected by prediction
    6002 0.069 1.00 679.0 0.060130 187.443486 14908 corrected by labels (control)
    9256 0.400 0.10 1939.0 0.398673 28.271429 15496 corrected by prediction
    9094 0.069 1.00 679.0 0.016429 686.039288 14580 corrected by prediction
    11329 0.100 0.10 1156.0 0.165504 68.101517 9384 corrected by prediction
    11012 1.000 0.01 1357.0 1.491715 7.555768 4462 corrected by prediction

    1000 rows × 7 columns

    # '$D$ in $\\frac{{\mu m^2}}{{s}}$'
    # '$\\tau_{{D}}$ in $ms$'
    x = '$D$ in $\\frac{{\mu m^2}}{{s}}$'
    g = sns.relplot(
              data=corr_scatter,
              x=x, y="Trace lengths",
              row="Traces used",
              hue="Simulated $D$",
              style="Simulated $D_{{clust}}$",
              kind="scatter",
              aspect=1.5,
              palette='colorblind').set(xscale='log')
    g.set_xlabels(x)
    g.tight_layout()
    g.fig.patch.set_facecolor('white')
    plt.show()
    

    We see that the distribution for \(D\) starts to spread out after rougly 10% of a trace has been deleted. The traces which had the most artifacts in them and thus were corrected most, were with a \(D_{clust}\) of 0.01. 2021-12-15_correlations_tracelengths-vs-diffrate-scatterplot.png

2.4.8 Learnings from run 1 and 2

  • The f1 metric does not work
  • I need to fix the fitting algorithm - as can be seen in <exp-…> the binning to 100um is not sufficient to correlate the plateau, thus we fit only the tail. This could be one reason why we get an average / median transit time of 0.5ms for uncorrupted traces, even though Pablo fitted 0.225ms in FOCUSpoint.
  • Boundary of correction method seems to be D=50, here correction by label and correction by prediction reduce the transit times too much, as in the achieved transit times are too low / diffrates are too fast.

2.5 exp-220120-correlate-ptu

2.5.1 Setup: Jupyter 1 on HPC compute node 1

  1. Setup tmux (#+CALL: setup-tmux[:session remote])
         
    sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:
    > ye53nis@ara-login01.rz.uni-jena.de's password:
  2. Request compute node via tmux
    cd /
    srun -p s_standard --time=7-10:00:00 --nodes=1 --ntasks-per-node=24 --mem=150000 --pty bash
    
    (base) [ye53nis@node305 /]$
    
  3. Branch out git branch exp-220120-correlate-ptu from main (done via magit) and make sure ou are on the correct branch
    cd /beegfs/ye53nis/drmed-git
    git checkout exp-220120-correlate-ptu
    
    (base) [ye53nis@node165 drmed-git]$ git checkout exp-220120-correlate-ptu
    Checking out files: 100% (147/147), done.
    M       src/nanosimpy
    Branch exp-220120-correlate-ptu set up to track remote branch exp-220120-correlate-ptu from origin.
    Switched to a new branch 'exp-220120-correlate-ptu'
    (base) [ye53nis@node165 drmed-git]$
    
  4. Make directory for experiment
    mkdir data/exp-220120-correlate-ptu/jupyter
    
  5. Customize the output folder using the following org-mode variable
    (setq org-babel-jupyter-resource-directory "./exp-220120-correlate-ptu/jupyter")
    
    ./exp-220120-correlate-ptu/jupyter
    
  6. Load conda environment, and start jupyter (#+CALL: jpt-tmux[:session jpmux])
    (tf) [ye53nis@node205 /]$ jupyter lab --no-browser --port=$PORT
    [I 2022-02-17 18:56:48.700 ServerApp] jupyterlab | extension was successfully linked.
    [I 2022-02-17 18:57:23.918 ServerApp] nbclassic | extension was successfully linked.
    [I 2022-02-17 18:57:27.274 LabApp] JupyterLab extension loaded from /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/jupyterlab
    [I 2022-02-17 18:57:27.275 LabApp] JupyterLab application directory is /home/ye53nis/.conda/envs/tf/share/jupyter/lab
    [I 2022-02-17 18:57:27.405 ServerApp] jupyterlab | extension was successfully loaded.
    [I 2022-02-17 18:57:28.130 ServerApp] nbclassic | extension was successfully loaded.
    [I 2022-02-17 18:57:28.130 ServerApp] Serving notebooks from local directory: /
    [I 2022-02-17 18:57:28.131 ServerApp] Jupyter Server 1.4.1 is running at:
    [I 2022-02-17 18:57:28.131 ServerApp] http://localhost:8889/lab?token=6f26a99296b37cdb05f3969fb347c51d49b9552c75ee7eb4
    [I 2022-02-17 18:57:28.131 ServerApp]  or http://127.0.0.1:8889/lab?token=6f26a99296b37cdb05f3969fb347c51d49b9552c75ee7eb4
    [I 2022-02-17 18:57:28.131 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
    [C 2022-02-17 18:57:28.499 ServerApp]
    
        To access the server, open this file in a browser:
            file:///home/ye53nis/.local/share/jupyter/runtime/jpserver-308527-open.html
        Or copy and paste one of these URLs:
            http://localhost:8889/lab?token=6f26a99296b37cdb05f3969fb347c51d49b9552c75ee7eb4
         or http://127.0.0.1:8889/lab?token=6f26a99296b37cdb05f3969fb347c51d49b9552c75ee7eb4
    
  7. Create ssh tunnel (#+CALL: ssh-tunnel(port="8889", node="node165"))
                         
    sh-5.1$ sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:              
    Warning: Permanently added 'node305,192.168.194.50' (ECDSA) to the list of known hosts.
    ye53nis@node305's password:                  
  8. I started a Python3 kernel using jupyter-server-list-kernels. Then I added the kernel ID to the :PROPERTIES: drawer of this (and following) subtrees.
    python3           03038b73-b2b5-49ce-a1dc-21afb6247d0f   a few seconds ago    starting   0
    
  9. Test and save metadata (including the python packages before the update in Experiment 2a):
    No of CPUs in system: 72
    No of CPUs the current process can use: 24
    load average: (6.14, 6.09, 6.06)
    os.uname():  posix.uname_result(sysname='Linux', nodename='node154', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
    PID of process: 59210
    RAM total: 199G, RAM used: 27G, RAM free: 85G
    the current directory: /
    My disk usage:
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/sda1             50G  3.8G   47G   8% /
    devtmpfs              94G     0   94G   0% /dev
    tmpfs                 94G  3.1M   94G   1% /dev/shm
    tmpfs                 94G  107M   94G   1% /run
    tmpfs                 94G     0   94G   0% /sys/fs/cgroup
    nfs01-ib:/home        80T   67T   14T  84% /home
    nfs01-ib:/cluster    2.0T  468G  1.6T  23% /cluster
    nfs03-ib:/pool/work  100T   79T   22T  79% /nfsdata
    nfs02-ib:/data01      88T   72T   16T  82% /data01
    /dev/sda3            6.0G  407M  5.6G   7% /var
    /dev/sda6            169G   11G  158G   7% /local
    /dev/sda5            2.0G   34M  2.0G   2% /tmp
    beegfs_nodev         524T  476T   49T  91% /beegfs
    tmpfs                 19G     0   19G   0% /run/user/67339# packages in environment at /home/ye53nis/.conda/envs/tf:
    #
    # Name                    Version                   Build  Channel
    _libgcc_mutex             0.1                        main
    _openmp_mutex             4.5                       1_gnu
    absl-py                   0.13.0                   pypi_0    pypi
    alembic                   1.4.1                    pypi_0    pypi
    anyio                     2.2.0            py39h06a4308_1
    argon2-cffi               20.1.0           py39h27cfd23_1
    asteval                   0.9.25                   pypi_0    pypi
    astunparse                1.6.3                    pypi_0    pypi
    async_generator           1.10               pyhd3eb1b0_0
    attrs                     21.2.0             pyhd3eb1b0_0
    babel                     2.9.1              pyhd3eb1b0_0
    backcall                  0.2.0              pyhd3eb1b0_0
    bleach                    3.3.1              pyhd3eb1b0_0
    brotlipy                  0.7.0           py39h27cfd23_1003
    ca-certificates           2021.7.5             h06a4308_1
    cachetools                4.2.2                    pypi_0    pypi
    certifi                   2021.5.30        py39h06a4308_0
    cffi                      1.14.6           py39h400218f_0
    chardet                   4.0.0           py39h06a4308_1003
    click                     8.0.1                    pypi_0    pypi
    cloudpickle               1.6.0                    pypi_0    pypi
    cryptography              3.4.7            py39hd23ed53_0
    cycler                    0.10.0                   pypi_0    pypi
    databricks-cli            0.14.3                   pypi_0    pypi
    decorator                 5.0.9              pyhd3eb1b0_0
    defusedxml                0.7.1              pyhd3eb1b0_0
    docker                    5.0.0                    pypi_0    pypi
    entrypoints               0.3              py39h06a4308_0
    fcsfiles                  2021.6.6                 pypi_0    pypi
    flask                     2.0.1                    pypi_0    pypi
    flatbuffers               1.12                     pypi_0    pypi
    future                    0.18.2                   pypi_0    pypi
    gast                      0.4.0                    pypi_0    pypi
    gitdb                     4.0.7                    pypi_0    pypi
    gitpython                 3.1.18                   pypi_0    pypi
    google-auth               1.34.0                   pypi_0    pypi
    google-auth-oauthlib      0.4.5                    pypi_0    pypi
    google-pasta              0.2.0                    pypi_0    pypi
    greenlet                  1.1.0                    pypi_0    pypi
    grpcio                    1.34.1                   pypi_0    pypi
    gunicorn                  20.1.0                   pypi_0    pypi
    h5py                      3.1.0                    pypi_0    pypi
    idna                      2.10               pyhd3eb1b0_0
    importlib-metadata        3.10.0           py39h06a4308_0
    importlib_metadata        3.10.0               hd3eb1b0_0
    ipykernel                 5.3.4            py39hb070fc8_0
    ipython                   7.22.0           py39hb070fc8_0
    ipython_genutils          0.2.0              pyhd3eb1b0_1
    itsdangerous              2.0.1                    pypi_0    pypi
    jedi                      0.17.2           py39h06a4308_1
    jinja2                    3.0.1              pyhd3eb1b0_0
    joblib                    1.0.1                    pypi_0    pypi
    json5                     0.9.6              pyhd3eb1b0_0
    jsonschema                3.2.0                      py_2
    jupyter-packaging         0.7.12             pyhd3eb1b0_0
    jupyter_client            6.1.12             pyhd3eb1b0_0
    jupyter_core              4.7.1            py39h06a4308_0
    jupyter_server            1.4.1            py39h06a4308_0
    jupyterlab                3.0.14             pyhd3eb1b0_1
    jupyterlab_pygments       0.1.2                      py_0
    jupyterlab_server         2.6.1              pyhd3eb1b0_0
    keras-nightly             2.5.0.dev2021032900          pypi_0    pypi
    keras-preprocessing       1.1.2                    pypi_0    pypi
    kiwisolver                1.3.1                    pypi_0    pypi
    ld_impl_linux-64          2.35.1               h7274673_9
    libffi                    3.3                  he6710b0_2
    libgcc-ng                 9.3.0               h5101ec6_17
    libgomp                   9.3.0               h5101ec6_17
    libsodium                 1.0.18               h7b6447c_0
    libstdcxx-ng              9.3.0               hd4cf53a_17
    lmfit                     1.0.2                    pypi_0    pypi
    mako                      1.1.4                    pypi_0    pypi
    markdown                  3.3.4                    pypi_0    pypi
    markupsafe                2.0.1            py39h27cfd23_0
    matplotlib                3.4.2                    pypi_0    pypi
    mistune                   0.8.4           py39h27cfd23_1000
    mlflow                    1.19.0                   pypi_0    pypi
    multipletau               0.3.3                    pypi_0    pypi
    nbclassic                 0.2.6              pyhd3eb1b0_0
    nbclient                  0.5.3              pyhd3eb1b0_0
    nbconvert                 6.1.0            py39h06a4308_0
    nbformat                  5.1.3              pyhd3eb1b0_0
    ncurses                   6.2                  he6710b0_1
    nest-asyncio              1.5.1              pyhd3eb1b0_0
    notebook                  6.4.0            py39h06a4308_0
    numpy                     1.19.5                   pypi_0    pypi
    oauthlib                  3.1.1                    pypi_0    pypi
    openssl                   1.1.1k               h27cfd23_0
    opt-einsum                3.3.0                    pypi_0    pypi
    packaging                 21.0               pyhd3eb1b0_0
    pandas                    1.3.1                    pypi_0    pypi
    pandocfilters             1.4.3            py39h06a4308_1
    parso                     0.7.0                      py_0
    pexpect                   4.8.0              pyhd3eb1b0_3
    pickleshare               0.7.5           pyhd3eb1b0_1003
    pillow                    8.3.1                    pypi_0    pypi
    pip                       21.1.3           py39h06a4308_0
    prometheus-flask-exporter 0.18.2                   pypi_0    pypi
    prometheus_client         0.11.0             pyhd3eb1b0_0
    prompt-toolkit            3.0.17             pyh06a4308_0
    protobuf                  3.17.3                   pypi_0    pypi
    ptyprocess                0.7.0              pyhd3eb1b0_2
    pyasn1                    0.4.8                    pypi_0    pypi
    pyasn1-modules            0.2.8                    pypi_0    pypi
    pycparser                 2.20                       py_2
    pygments                  2.9.0              pyhd3eb1b0_0
    pyopenssl                 20.0.1             pyhd3eb1b0_1
    pyparsing                 2.4.7              pyhd3eb1b0_0
    pyrsistent                0.18.0           py39h7f8727e_0
    pysocks                   1.7.1            py39h06a4308_0
    python                    3.9.5                h12debd9_4
    python-dateutil           2.8.2              pyhd3eb1b0_0
    python-editor             1.0.4                    pypi_0    pypi
    pytz                      2021.1             pyhd3eb1b0_0
    pyyaml                    5.4.1                    pypi_0    pypi
    pyzmq                     20.0.0           py39h2531618_1
    querystring-parser        1.2.4                    pypi_0    pypi
    readline                  8.1                  h27cfd23_0
    requests                  2.25.1             pyhd3eb1b0_0
    requests-oauthlib         1.3.0                    pypi_0    pypi
    rsa                       4.7.2                    pypi_0    pypi
    scikit-learn              0.24.2                   pypi_0    pypi
    scipy                     1.7.0                    pypi_0    pypi
    seaborn                   0.11.1                   pypi_0    pypi
    send2trash                1.5.0              pyhd3eb1b0_1
    setuptools                52.0.0           py39h06a4308_0
    six                       1.15.0                   pypi_0    pypi
    smmap                     4.0.0                    pypi_0    pypi
    sniffio                   1.2.0            py39h06a4308_1
    sqlalchemy                1.4.22                   pypi_0    pypi
    sqlite                    3.36.0               hc218d9a_0
    sqlparse                  0.4.1                    pypi_0    pypi
    tabulate                  0.8.9                    pypi_0    pypi
    tensorboard               2.5.0                    pypi_0    pypi
    tensorboard-data-server   0.6.1                    pypi_0    pypi
    tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
    tensorflow                2.5.0                    pypi_0    pypi
    tensorflow-estimator      2.5.0                    pypi_0    pypi
    termcolor                 1.1.0                    pypi_0    pypi
    terminado                 0.9.4            py39h06a4308_0
    testpath                  0.5.0              pyhd3eb1b0_0
    threadpoolctl             2.2.0                    pypi_0    pypi
    tifffile                  2021.7.30                pypi_0    pypi
    tk                        8.6.10               hbc83047_0
    tornado                   6.1              py39h27cfd23_0
    traitlets                 5.0.5              pyhd3eb1b0_0
    typing-extensions         3.7.4.3                  pypi_0    pypi
    tzdata                    2021a                h52ac0ba_0
    uncertainties             3.1.6                    pypi_0    pypi
    urllib3                   1.26.6             pyhd3eb1b0_1
    wcwidth                   0.2.5                      py_0
    webencodings              0.5.1            py39h06a4308_1
    websocket-client          1.1.0                    pypi_0    pypi
    werkzeug                  2.0.1                    pypi_0    pypi
    wheel                     0.36.2             pyhd3eb1b0_0
    wrapt                     1.12.1                   pypi_0    pypi
    xz                        5.2.5                h7b6447c_0
    zeromq                    4.3.4                h2531618_0
    zipp                      3.5.0              pyhd3eb1b0_0
    zlib                      1.2.11               h7b6447c_3
    
    Note: you may need to restart the kernel to use updated packages.
    {'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
     'SLURM_NODELIST': 'node154',
     'SLURM_JOB_NAME': 'bash',
     'XDG_SESSION_ID': '44301',
     'SLURMD_NODENAME': 'node154',
     'SLURM_TOPOLOGY_ADDR': 'node154',
     'SLURM_NTASKS_PER_NODE': '24',
     'HOSTNAME': 'login01',
     'SLURM_PRIO_PROCESS': '0',
     'SLURM_SRUN_COMM_PORT': '41523',
     'SHELL': '/bin/bash',
     'TERM': 'xterm-color',
     'SLURM_JOB_QOS': 'qstand',
     'SLURM_PTY_WIN_ROW': '34',
     'HISTSIZE': '1000',
     'TMPDIR': '/tmp',
     'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
     'SSH_CLIENT': '10.231.181.128 49370 22',
     'CONDA_SHLVL': '2',
     'CONDA_PROMPT_MODIFIER': '(tf) ',
     'QTDIR': '/usr/lib64/qt-3.3',
     'QTINC': '/usr/lib64/qt-3.3/include',
     'SSH_TTY': '/dev/pts/79',
     'NO_PROXY': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001',
     'QT_GRAPHICSSYSTEM_CHECKED': '1',
     'SLURM_NNODES': '1',
     'USER': 'ye53nis',
     'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
     'CONDA_EXE': '/cluster/miniconda3/bin/conda',
     'SLURM_STEP_NUM_NODES': '1',
     'SLURM_JOBID': '1608805',
     'SRUN_DEBUG': '3',
     'FTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'ftp_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_NTASKS': '24',
     'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
     'SLURM_STEP_ID': '0',
     'TMUX': '/tmp/tmux-67339/default,20557,4',
     '_CE_CONDA': '',
     'CONDA_PREFIX_1': '/cluster/miniconda3',
     'SLURM_STEP_LAUNCHER_PORT': '41523',
     'SLURM_TASKS_PER_NODE': '24',
     'MAIL': '/var/spool/mail/ye53nis',
     'PATH': '/home/ye53nis/.conda/envs/tf/bin:/home/lex/Programme/miniconda3/envs/tf-nightly-lab/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
     'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
     'SLURM_JOB_ID': '1608805',
     'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tf',
     'SLURM_JOB_USER': 'ye53nis',
     'SLURM_STEPID': '0',
     'PWD': '/',
     'SLURM_SRUN_COMM_HOST': '192.168.192.5',
     'LANG': 'en_US.UTF-8',
     'SLURM_PTY_WIN_COL': '236',
     'SLURM_UMASK': '0022',
     'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
     'SLURM_JOB_UID': '67339',
     'LOADEDMODULES': '',
     'SLURM_NODEID': '0',
     'TMUX_PANE': '%4',
     'SLURM_SUBMIT_DIR': '/',
     'SLURM_TASK_PID': '57304',
     'SLURM_NPROCS': '24',
     'SLURM_CPUS_ON_NODE': '24',
     'SLURM_DISTRIBUTION': 'block',
     'HTTPS_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'https_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_PROCID': '0',
     'HISTCONTROL': 'ignoredups',
     '_CE_M': '',
     'SLURM_JOB_NODELIST': 'node154',
     'SLURM_PTY_PORT': '38987',
     'HOME': '/home/ye53nis',
     'SHLVL': '3',
     'SLURM_LOCALID': '0',
     'SLURM_JOB_GID': '13280',
     'SLURM_JOB_CPUS_PER_NODE': '24',
     'SLURM_CLUSTER_NAME': 'hpc',
     'no_proxy': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001',
     'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
     'SLURM_SUBMIT_HOST': 'login01',
     'HTTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_JOB_PARTITION': 's_standard',
     'MATHEMATICA_HOME': '/cluster/apps/mathematica/11.3',
     'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
     'LOGNAME': 'ye53nis',
     'SLURM_STEP_NUM_TASKS': '24',
     'QTLIB': '/usr/lib64/qt-3.3/lib',
     'SLURM_JOB_ACCOUNT': 'iaob',
     'SLURM_JOB_NUM_NODES': '1',
     'MODULESHOME': '/usr/share/Modules',
     'CONDA_DEFAULT_ENV': 'tf',
     'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
     'SLURM_STEP_TASKS_PER_NODE': '24',
     'PORT': '8889',
     'SLURM_STEP_NODELIST': 'node154',
     'DISPLAY': ':0',
     'XDG_RUNTIME_DIR': '',
     'XAUTHORITY': '/home/lex/.Xauthority',
     'BASH_FUNC_module()': '() {  eval `/usr/bin/modulecmd bash $*`\n}',
     '_': '/home/ye53nis/.conda/envs/tf/bin/jupyter',
     'JPY_PARENT_PID': '58148',
     'CLICOLOR': '1',
     'PAGER': 'cat',
     'GIT_PAGER': 'cat',
     'MPLBACKEND': 'module://ipykernel.pylab.backend_inline'}
    
  10. Test and save metadata after update in Experiment 2a
    No of CPUs in system: 72
    No of CPUs the current process can use: 24
    load average: (8.76, 8.8, 8.92)
    os.uname():  posix.uname_result(sysname='Linux', nodename='node154', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
    PID of process: 151057
    RAM total: 199G, RAM used: 45G, RAM free: 66G
    the current directory: /beegfs/ye53nis/drmed-git
    My disk usage:
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/sda1             50G  3.8G   47G   8% /
    devtmpfs              94G     0   94G   0% /dev
    tmpfs                 94G  3.1M   94G   1% /dev/shm
    tmpfs                 94G  107M   94G   1% /run
    tmpfs                 94G     0   94G   0% /sys/fs/cgroup
    nfs01-ib:/home        80T   67T   14T  84% /home
    nfs01-ib:/cluster    2.0T  468G  1.6T  23% /cluster
    nfs03-ib:/pool/work  100T   79T   22T  79% /nfsdata
    nfs02-ib:/data01      88T   72T   16T  82% /data01
    /dev/sda3            6.0G  406M  5.6G   7% /var
    /dev/sda6            169G   11G  158G   7% /local
    /dev/sda5            2.0G   34M  2.0G   2% /tmp
    beegfs_nodev         524T  476T   49T  91% /beegfs
    tmpfs                 19G     0   19G   0% /run/user/67339# packages in environment at /home/ye53nis/.conda/envs/tf:
    #
    # Name                    Version                   Build  Channel
    _libgcc_mutex             0.1                        main
    _openmp_mutex             4.5                       1_gnu
    absl-py                   1.0.0                    pypi_0    pypi
    alembic                   1.7.6                    pypi_0    pypi
    anyio                     2.2.0            py39h06a4308_1
    argon2-cffi               20.1.0           py39h27cfd23_1
    asteval                   0.9.26                   pypi_0    pypi
    astunparse                1.6.3                    pypi_0    pypi
    async_generator           1.10               pyhd3eb1b0_0
    attrs                     21.4.0             pyhd3eb1b0_0
    babel                     2.9.1              pyhd3eb1b0_0
    backcall                  0.2.0              pyhd3eb1b0_0
    bleach                    4.1.0              pyhd3eb1b0_0
    brotlipy                  0.7.0           py39h27cfd23_1003
    ca-certificates           2021.10.26           h06a4308_2
    cachetools                5.0.0                    pypi_0    pypi
    certifi                   2021.10.8        py39h06a4308_2
    cffi                      1.15.0           py39hd667e15_1
    charset-normalizer        2.0.4              pyhd3eb1b0_0
    click                     8.0.3                    pypi_0    pypi
    cloudpickle               2.0.0                    pypi_0    pypi
    cryptography              36.0.0           py39h9ce1e76_0
    cycler                    0.11.0                   pypi_0    pypi
    cython                    0.29.27                  pypi_0    pypi
    databricks-cli            0.16.4                   pypi_0    pypi
    debugpy                   1.5.1            py39h295c915_0
    decorator                 5.1.1              pyhd3eb1b0_0
    defusedxml                0.7.1              pyhd3eb1b0_0
    docker                    5.0.3                    pypi_0    pypi
    entrypoints               0.3              py39h06a4308_0
    fcsfiles                  2022.2.2                 pypi_0    pypi
    flask                     2.0.2                    pypi_0    pypi
    flatbuffers               2.0                      pypi_0    pypi
    fonttools                 4.29.1                   pypi_0    pypi
    future                    0.18.2                   pypi_0    pypi
    gast                      0.5.3                    pypi_0    pypi
    gitdb                     4.0.9                    pypi_0    pypi
    gitpython                 3.1.26                   pypi_0    pypi
    google-auth               2.6.0                    pypi_0    pypi
    google-auth-oauthlib      0.4.6                    pypi_0    pypi
    google-pasta              0.2.0                    pypi_0    pypi
    greenlet                  1.1.2                    pypi_0    pypi
    grpcio                    1.43.0                   pypi_0    pypi
    gunicorn                  20.1.0                   pypi_0    pypi
    h5py                      3.6.0                    pypi_0    pypi
    idna                      3.3                pyhd3eb1b0_0
    importlib-metadata        4.8.2            py39h06a4308_0
    importlib_metadata        4.8.2                hd3eb1b0_0
    ipykernel                 6.4.1            py39h06a4308_1
    ipython                   7.31.1           py39h06a4308_0
    ipython_genutils          0.2.0              pyhd3eb1b0_1
    itsdangerous              2.0.1                    pypi_0    pypi
    jedi                      0.18.1           py39h06a4308_1
    jinja2                    3.0.2              pyhd3eb1b0_0
    joblib                    1.1.0                    pypi_0    pypi
    json5                     0.9.6              pyhd3eb1b0_0
    jsonschema                3.2.0              pyhd3eb1b0_2
    jupyter_client            7.1.2              pyhd3eb1b0_0
    jupyter_core              4.9.1            py39h06a4308_0
    jupyter_server            1.4.1            py39h06a4308_0
    jupyterlab                3.2.1              pyhd3eb1b0_1
    jupyterlab_pygments       0.1.2                      py_0
    jupyterlab_server         2.10.2             pyhd3eb1b0_1
    keras                     2.8.0                    pypi_0    pypi
    keras-preprocessing       1.1.2                    pypi_0    pypi
    kiwisolver                1.3.2                    pypi_0    pypi
    ld_impl_linux-64          2.35.1               h7274673_9
    libclang                  13.0.0                   pypi_0    pypi
    libffi                    3.3                  he6710b0_2
    libgcc-ng                 9.3.0               h5101ec6_17
    libgomp                   9.3.0               h5101ec6_17
    libsodium                 1.0.18               h7b6447c_0
    libstdcxx-ng              9.3.0               hd4cf53a_17
    lmfit                     1.0.3                    pypi_0    pypi
    mako                      1.1.6                    pypi_0    pypi
    markdown                  3.3.6                    pypi_0    pypi
    markupsafe                2.0.1            py39h27cfd23_0
    matplotlib                3.5.1                    pypi_0    pypi
    matplotlib-inline         0.1.2              pyhd3eb1b0_2
    mistune                   0.8.4           py39h27cfd23_1000
    mlflow                    1.23.1                   pypi_0    pypi
    multipletau               0.3.3                    pypi_0    pypi
    nbclassic                 0.2.6              pyhd3eb1b0_0
    nbclient                  0.5.3              pyhd3eb1b0_0
    nbconvert                 6.3.0            py39h06a4308_0
    nbformat                  5.1.3              pyhd3eb1b0_0
    ncurses                   6.3                  h7f8727e_2
    nest-asyncio              1.5.1              pyhd3eb1b0_0
    notebook                  6.4.6            py39h06a4308_0
    numpy                     1.22.2                   pypi_0    pypi
    oauthlib                  3.2.0                    pypi_0    pypi
    openssl                   1.1.1m               h7f8727e_0
    opt-einsum                3.3.0                    pypi_0    pypi
    packaging                 21.3               pyhd3eb1b0_0
    pandas                    1.4.0                    pypi_0    pypi
    pandocfilters             1.5.0              pyhd3eb1b0_0
    parso                     0.8.3              pyhd3eb1b0_0
    pexpect                   4.8.0              pyhd3eb1b0_3
    pickleshare               0.7.5           pyhd3eb1b0_1003
    pillow                    9.0.1                    pypi_0    pypi
    pip                       21.2.4           py39h06a4308_0
    prometheus-flask-exporter 0.18.7                   pypi_0    pypi
    prometheus_client         0.13.1             pyhd3eb1b0_0
    prompt-toolkit            3.0.20             pyhd3eb1b0_0
    protobuf                  3.19.4                   pypi_0    pypi
    ptyprocess                0.7.0              pyhd3eb1b0_2
    pyasn1                    0.4.8                    pypi_0    pypi
    pyasn1-modules            0.2.8                    pypi_0    pypi
    pycparser                 2.21               pyhd3eb1b0_0
    pygments                  2.11.2             pyhd3eb1b0_0
    pyopenssl                 22.0.0             pyhd3eb1b0_0
    pyparsing                 3.0.4              pyhd3eb1b0_0
    pyrsistent                0.18.0           py39heee7806_0
    pysocks                   1.7.1            py39h06a4308_0
    python                    3.9.7                h12debd9_1
    python-dateutil           2.8.2              pyhd3eb1b0_0
    pytz                      2021.3             pyhd3eb1b0_0
    pyyaml                    6.0                      pypi_0    pypi
    pyzmq                     22.3.0           py39h295c915_2
    querystring-parser        1.2.4                    pypi_0    pypi
    readline                  8.1.2                h7f8727e_1
    requests                  2.27.1             pyhd3eb1b0_0
    requests-oauthlib         1.3.1                    pypi_0    pypi
    rsa                       4.8                      pypi_0    pypi
    scikit-learn              1.0.2                    pypi_0    pypi
    scipy                     1.8.0                    pypi_0    pypi
    seaborn                   0.11.2                   pypi_0    pypi
    send2trash                1.8.0              pyhd3eb1b0_1
    setuptools                58.0.4           py39h06a4308_0
    six                       1.16.0             pyhd3eb1b0_0
    smmap                     5.0.0                    pypi_0    pypi
    sniffio                   1.2.0            py39h06a4308_1
    sqlalchemy                1.4.31                   pypi_0    pypi
    sqlite                    3.37.2               hc218d9a_0
    sqlparse                  0.4.2                    pypi_0    pypi
    tabulate                  0.8.9                    pypi_0    pypi
    tensorboard               2.8.0                    pypi_0    pypi
    tensorboard-data-server   0.6.1                    pypi_0    pypi
    tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
    tensorflow                2.8.0                    pypi_0    pypi
    tensorflow-io-gcs-filesystem 0.24.0                   pypi_0    pypi
    termcolor                 1.1.0                    pypi_0    pypi
    terminado                 0.9.4            py39h06a4308_0
    testpath                  0.5.0              pyhd3eb1b0_0
    tf-estimator-nightly      2.8.0.dev2021122109          pypi_0    pypi
    threadpoolctl             3.1.0                    pypi_0    pypi
    tk                        8.6.11               h1ccaba5_0
    tornado                   6.1              py39h27cfd23_0
    traitlets                 5.1.1              pyhd3eb1b0_0
    typing-extensions         4.0.1                    pypi_0    pypi
    tzdata                    2021e                hda174b7_0
    uncertainties             3.1.6                    pypi_0    pypi
    urllib3                   1.26.8             pyhd3eb1b0_0
    wcwidth                   0.2.5              pyhd3eb1b0_0
    webencodings              0.5.1            py39h06a4308_1
    websocket-client          1.2.3                    pypi_0    pypi
    werkzeug                  2.0.3                    pypi_0    pypi
    wheel                     0.37.1             pyhd3eb1b0_0
    wrapt                     1.13.3                   pypi_0    pypi
    xz                        5.2.5                h7b6447c_0
    zeromq                    4.3.4                h2531618_0
    zipp                      3.7.0              pyhd3eb1b0_0
    zlib                      1.2.11               h7f8727e_4
    
    Note: you may need to restart the kernel to use updated packages.
    {'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
     'SLURM_NODELIST': 'node154',
     'SLURM_JOB_NAME': 'bash',
     'XDG_SESSION_ID': '44301',
     'SLURMD_NODENAME': 'node154',
     'SLURM_TOPOLOGY_ADDR': 'node154',
     'SLURM_NTASKS_PER_NODE': '24',
     'HOSTNAME': 'login01',
     'SLURM_PRIO_PROCESS': '0',
     'SLURM_SRUN_COMM_PORT': '41523',
     'SHELL': '/bin/bash',
     'TERM': 'xterm-color',
     'SLURM_JOB_QOS': 'qstand',
     'SLURM_PTY_WIN_ROW': '34',
     'HISTSIZE': '1000',
     'TMPDIR': '/tmp',
     'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
     'SSH_CLIENT': '10.231.181.128 49370 22',
     'CONDA_SHLVL': '2',
     'CONDA_PROMPT_MODIFIER': '(tf) ',
     'QTDIR': '/usr/lib64/qt-3.3',
     'QTINC': '/usr/lib64/qt-3.3/include',
     'SSH_TTY': '/dev/pts/79',
     'NO_PROXY': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001',
     'QT_GRAPHICSSYSTEM_CHECKED': '1',
     'SLURM_NNODES': '1',
     'USER': 'ye53nis',
     'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
     'CONDA_EXE': '/cluster/miniconda3/bin/conda',
     'SLURM_STEP_NUM_NODES': '1',
     'SLURM_JOBID': '1608805',
     'SRUN_DEBUG': '3',
     'FTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'ftp_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_NTASKS': '24',
     'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
     'SLURM_STEP_ID': '0',
     'TMUX': '/tmp/tmux-67339/default,20557,4',
     '_CE_CONDA': '',
     'CONDA_PREFIX_1': '/cluster/miniconda3',
     'SLURM_STEP_LAUNCHER_PORT': '41523',
     'SLURM_TASKS_PER_NODE': '24',
     'MAIL': '/var/spool/mail/ye53nis',
     'PATH': '/home/ye53nis/.conda/envs/tf/bin:/home/lex/Programme/miniconda3/envs/tf-nightly-lab/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
     'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
     'SLURM_JOB_ID': '1608805',
     'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tf',
     'SLURM_JOB_USER': 'ye53nis',
     'SLURM_STEPID': '0',
     'PWD': '/',
     'SLURM_SRUN_COMM_HOST': '192.168.192.5',
     'LANG': 'en_US.UTF-8',
     'SLURM_PTY_WIN_COL': '236',
     'SLURM_UMASK': '0022',
     'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
     'SLURM_JOB_UID': '67339',
     'LOADEDMODULES': '',
     'SLURM_NODEID': '0',
     'TMUX_PANE': '%4',
     'SLURM_SUBMIT_DIR': '/',
     'SLURM_TASK_PID': '57304',
     'SLURM_NPROCS': '24',
     'SLURM_CPUS_ON_NODE': '24',
     'SLURM_DISTRIBUTION': 'block',
     'HTTPS_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'https_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_PROCID': '0',
     'HISTCONTROL': 'ignoredups',
     '_CE_M': '',
     'SLURM_JOB_NODELIST': 'node154',
     'SLURM_PTY_PORT': '38987',
     'HOME': '/home/ye53nis',
     'SHLVL': '3',
     'SLURM_LOCALID': '0',
     'SLURM_JOB_GID': '13280',
     'SLURM_JOB_CPUS_PER_NODE': '24',
     'SLURM_CLUSTER_NAME': 'hpc',
     'no_proxy': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001',
     'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
     'SLURM_SUBMIT_HOST': 'login01',
     'HTTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_JOB_PARTITION': 's_standard',
     'MATHEMATICA_HOME': '/cluster/apps/mathematica/11.3',
     'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
     'LOGNAME': 'ye53nis',
     'SLURM_STEP_NUM_TASKS': '24',
     'QTLIB': '/usr/lib64/qt-3.3/lib',
     'SLURM_JOB_ACCOUNT': 'iaob',
     'SLURM_JOB_NUM_NODES': '1',
     'MODULESHOME': '/usr/share/Modules',
     'CONDA_DEFAULT_ENV': 'tf',
     'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
     'SLURM_STEP_TASKS_PER_NODE': '24',
     'PORT': '8889',
     'SLURM_STEP_NODELIST': 'node154',
     'DISPLAY': ':0',
     'XDG_RUNTIME_DIR': '',
     'XAUTHORITY': '/home/lex/.Xauthority',
     'BASH_FUNC_module()': '() {  eval `/usr/bin/modulecmd bash $*`\n}',
     '_': '/home/ye53nis/.conda/envs/tf/bin/jupyter',
     'JPY_PARENT_PID': '58148',
     'PYDEVD_USE_FRAME_EVAL': 'NO',
     'CLICOLOR': '1',
     'PAGER': 'cat',
     'GIT_PAGER': 'cat',
     'MPLBACKEND': 'module://matplotlib_inline.backend_inline',
     'TF2_BEHAVIOR': '1',
     'KMP_DUPLICATE_LIB_OK': 'True',
     'KMP_INIT_AT_FORK': 'FALSE'}
    
  11. Test and save metadata for Analysis 4 (190327_detectordropout needs a lot of RAM)
    No of CPUs in system: 72
    No of CPUs the current process can use: 24
    load average: (0.0, 0.01, 0.05)
    os.uname():  posix.uname_result(sysname='Linux', nodename='node305', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
    PID of process: 215732
    RAM total: 199G, RAM used: 4.9G, RAM free: 110G
    the current directory: /
    My disk usage:
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/sda1             50G  3.8G   47G   8% /
    devtmpfs              94G     0   94G   0% /dev
    tmpfs                 94G  5.6G   89G   6% /dev/shm
    tmpfs                 94G  155M   94G   1% /run
    tmpfs                 94G     0   94G   0% /sys/fs/cgroup
    nfs01-ib:/home        80T   68T   13T  85% /home
    nfs03-ib:/pool/work  100T   71T   29T  71% /nfsdata
    nfs01-ib:/cluster    2.0T  468G  1.6T  23% /cluster
    nfs02-ib:/data01      88T   72T   16T  82% /data01
    /dev/sda6            169G  2.6G  166G   2% /local
    /dev/sda5            2.0G   34M  2.0G   2% /tmp
    /dev/sda3            6.0G  419M  5.6G   7% /var
    beegfs_nodev         524T  491T   34T  94% /beegfs
    tmpfs                 19G     0   19G   0% /run/user/67339# packages in environment at /home/ye53nis/.conda/envs/tf:
    #
    # Name                    Version                   Build  Channel
    _libgcc_mutex             0.1                        main
    _openmp_mutex             4.5                       1_gnu
    absl-py                   1.0.0                    pypi_0    pypi
    alembic                   1.7.6                    pypi_0    pypi
    anyio                     2.2.0            py39h06a4308_1
    argon2-cffi               20.1.0           py39h27cfd23_1
    asteval                   0.9.26                   pypi_0    pypi
    astunparse                1.6.3                    pypi_0    pypi
    async_generator           1.10               pyhd3eb1b0_0
    attrs                     21.4.0             pyhd3eb1b0_0
    babel                     2.9.1              pyhd3eb1b0_0
    backcall                  0.2.0              pyhd3eb1b0_0
    bleach                    4.1.0              pyhd3eb1b0_0
    brotlipy                  0.7.0           py39h27cfd23_1003
    ca-certificates           2021.10.26           h06a4308_2
    cachetools                5.0.0                    pypi_0    pypi
    certifi                   2021.10.8        py39h06a4308_2
    cffi                      1.15.0           py39hd667e15_1
    charset-normalizer        2.0.4              pyhd3eb1b0_0
    click                     8.0.3                    pypi_0    pypi
    cloudpickle               2.0.0                    pypi_0    pypi
    cryptography              36.0.0           py39h9ce1e76_0
    cycler                    0.11.0                   pypi_0    pypi
    cython                    0.29.27                  pypi_0    pypi
    databricks-cli            0.16.4                   pypi_0    pypi
    debugpy                   1.5.1            py39h295c915_0
    decorator                 5.1.1              pyhd3eb1b0_0
    defusedxml                0.7.1              pyhd3eb1b0_0
    docker                    5.0.3                    pypi_0    pypi
    entrypoints               0.3              py39h06a4308_0
    fcsfiles                  2022.2.2                 pypi_0    pypi
    flask                     2.0.2                    pypi_0    pypi
    flatbuffers               2.0                      pypi_0    pypi
    fonttools                 4.29.1                   pypi_0    pypi
    future                    0.18.2                   pypi_0    pypi
    gast                      0.5.3                    pypi_0    pypi
    gitdb                     4.0.9                    pypi_0    pypi
    gitpython                 3.1.26                   pypi_0    pypi
    google-auth               2.6.0                    pypi_0    pypi
    google-auth-oauthlib      0.4.6                    pypi_0    pypi
    google-pasta              0.2.0                    pypi_0    pypi
    greenlet                  1.1.2                    pypi_0    pypi
    grpcio                    1.43.0                   pypi_0    pypi
    gunicorn                  20.1.0                   pypi_0    pypi
    h5py                      3.6.0                    pypi_0    pypi
    idna                      3.3                pyhd3eb1b0_0
    importlib-metadata        4.8.2            py39h06a4308_0
    importlib_metadata        4.8.2                hd3eb1b0_0
    ipykernel                 6.4.1            py39h06a4308_1
    ipython                   7.31.1           py39h06a4308_0
    ipython_genutils          0.2.0              pyhd3eb1b0_1
    itsdangerous              2.0.1                    pypi_0    pypi
    jedi                      0.18.1           py39h06a4308_1
    jinja2                    3.0.2              pyhd3eb1b0_0
    joblib                    1.1.0                    pypi_0    pypi
    json5                     0.9.6              pyhd3eb1b0_0
    jsonschema                3.2.0              pyhd3eb1b0_2
    jupyter_client            7.1.2              pyhd3eb1b0_0
    jupyter_core              4.9.1            py39h06a4308_0
    jupyter_server            1.4.1            py39h06a4308_0
    jupyterlab                3.2.1              pyhd3eb1b0_1
    jupyterlab_pygments       0.1.2                      py_0
    jupyterlab_server         2.10.2             pyhd3eb1b0_1
    keras                     2.8.0                    pypi_0    pypi
    keras-preprocessing       1.1.2                    pypi_0    pypi
    kiwisolver                1.3.2                    pypi_0    pypi
    ld_impl_linux-64          2.35.1               h7274673_9
    libclang                  13.0.0                   pypi_0    pypi
    libffi                    3.3                  he6710b0_2
    libgcc-ng                 9.3.0               h5101ec6_17
    libgomp                   9.3.0               h5101ec6_17
    libsodium                 1.0.18               h7b6447c_0
    libstdcxx-ng              9.3.0               hd4cf53a_17
    lmfit                     1.0.3                    pypi_0    pypi
    mako                      1.1.6                    pypi_0    pypi
    markdown                  3.3.6                    pypi_0    pypi
    markupsafe                2.0.1            py39h27cfd23_0
    matplotlib                3.5.1                    pypi_0    pypi
    matplotlib-inline         0.1.2              pyhd3eb1b0_2
    mistune                   0.8.4           py39h27cfd23_1000
    mlflow                    1.23.1                   pypi_0    pypi
    multipletau               0.3.3                    pypi_0    pypi
    nbclassic                 0.2.6              pyhd3eb1b0_0
    nbclient                  0.5.3              pyhd3eb1b0_0
    nbconvert                 6.3.0            py39h06a4308_0
    nbformat                  5.1.3              pyhd3eb1b0_0
    ncurses                   6.3                  h7f8727e_2
    nest-asyncio              1.5.1              pyhd3eb1b0_0
    notebook                  6.4.6            py39h06a4308_0
    numpy                     1.22.2                   pypi_0    pypi
    oauthlib                  3.2.0                    pypi_0    pypi
    openssl                   1.1.1m               h7f8727e_0
    opt-einsum                3.3.0                    pypi_0    pypi
    packaging                 21.3               pyhd3eb1b0_0
    pandas                    1.4.0                    pypi_0    pypi
    pandocfilters             1.5.0              pyhd3eb1b0_0
    parso                     0.8.3              pyhd3eb1b0_0
    pexpect                   4.8.0              pyhd3eb1b0_3
    pickleshare               0.7.5           pyhd3eb1b0_1003
    pillow                    9.0.1                    pypi_0    pypi
    pip                       21.2.4           py39h06a4308_0
    prometheus-flask-exporter 0.18.7                   pypi_0    pypi
    prometheus_client         0.13.1             pyhd3eb1b0_0
    prompt-toolkit            3.0.20             pyhd3eb1b0_0
    protobuf                  3.19.4                   pypi_0    pypi
    ptyprocess                0.7.0              pyhd3eb1b0_2
    pyasn1                    0.4.8                    pypi_0    pypi
    pyasn1-modules            0.2.8                    pypi_0    pypi
    pycparser                 2.21               pyhd3eb1b0_0
    pygments                  2.11.2             pyhd3eb1b0_0
    pyopenssl                 22.0.0             pyhd3eb1b0_0
    pyparsing                 3.0.4              pyhd3eb1b0_0
    pyrsistent                0.18.0           py39heee7806_0
    pysocks                   1.7.1            py39h06a4308_0
    python                    3.9.7                h12debd9_1
    python-dateutil           2.8.2              pyhd3eb1b0_0
    pytz                      2021.3             pyhd3eb1b0_0
    pyyaml                    6.0                      pypi_0    pypi
    pyzmq                     22.3.0           py39h295c915_2
    querystring-parser        1.2.4                    pypi_0    pypi
    readline                  8.1.2                h7f8727e_1
    requests                  2.27.1             pyhd3eb1b0_0
    requests-oauthlib         1.3.1                    pypi_0    pypi
    rsa                       4.8                      pypi_0    pypi
    scikit-learn              1.0.2                    pypi_0    pypi
    scipy                     1.8.0                    pypi_0    pypi
    seaborn                   0.11.2                   pypi_0    pypi
    send2trash                1.8.0              pyhd3eb1b0_1
    setuptools                58.0.4           py39h06a4308_0
    six                       1.16.0             pyhd3eb1b0_0
    smmap                     5.0.0                    pypi_0    pypi
    sniffio                   1.2.0            py39h06a4308_1
    sqlalchemy                1.4.31                   pypi_0    pypi
    sqlite                    3.37.2               hc218d9a_0
    sqlparse                  0.4.2                    pypi_0    pypi
    tabulate                  0.8.9                    pypi_0    pypi
    tensorboard               2.8.0                    pypi_0    pypi
    tensorboard-data-server   0.6.1                    pypi_0    pypi
    tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
    tensorflow                2.8.0                    pypi_0    pypi
    tensorflow-io-gcs-filesystem 0.24.0                   pypi_0    pypi
    termcolor                 1.1.0                    pypi_0    pypi
    terminado                 0.9.4            py39h06a4308_0
    testpath                  0.5.0              pyhd3eb1b0_0
    tf-estimator-nightly      2.8.0.dev2021122109          pypi_0    pypi
    threadpoolctl             3.1.0                    pypi_0    pypi
    tk                        8.6.11               h1ccaba5_0
    tornado                   6.1              py39h27cfd23_0
    traitlets                 5.1.1              pyhd3eb1b0_0
    typing-extensions         4.0.1                    pypi_0    pypi
    tzdata                    2021e                hda174b7_0
    uncertainties             3.1.6                    pypi_0    pypi
    urllib3                   1.26.8             pyhd3eb1b0_0
    wcwidth                   0.2.5              pyhd3eb1b0_0
    webencodings              0.5.1            py39h06a4308_1
    websocket-client          1.2.3                    pypi_0    pypi
    werkzeug                  2.0.3                    pypi_0    pypi
    wheel                     0.37.1             pyhd3eb1b0_0
    wrapt                     1.13.3                   pypi_0    pypi
    xz                        5.2.5                h7b6447c_0
    zeromq                    4.3.4                h2531618_0
    zipp                      3.7.0              pyhd3eb1b0_0
    zlib                      1.2.11               h7f8727e_4
    
    Note: you may need to restart the kernel to use updated packages.
    {'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
     'SLURM_NODELIST': 'node305',
     'SLURM_JOB_NAME': 'bash',
     'XDG_SESSION_ID': '44301',
     'SLURMD_NODENAME': 'node305',
     'SLURM_TOPOLOGY_ADDR': 'node305',
     'SLURM_NTASKS_PER_NODE': '24',
     'HOSTNAME': 'login01',
     'SLURM_PRIO_PROCESS': '0',
     'SLURM_SRUN_COMM_PORT': '44912',
     'SHELL': '/bin/bash',
     'TERM': 'xterm-color',
     'SLURM_JOB_QOS': 'qstand',
     'SLURM_PTY_WIN_ROW': '34',
     'HISTSIZE': '1000',
     'TMPDIR': '/tmp',
     'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
     'SSH_CLIENT': '10.231.181.128 49370 22',
     'CONDA_SHLVL': '2',
     'CONDA_PROMPT_MODIFIER': '(tf) ',
     'QTDIR': '/usr/lib64/qt-3.3',
     'QTINC': '/usr/lib64/qt-3.3/include',
     'SSH_TTY': '/dev/pts/79',
     'NO_PROXY': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001',
     'QT_GRAPHICSSYSTEM_CHECKED': '1',
     'SLURM_NNODES': '1',
     'USER': 'ye53nis',
     'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
     'CONDA_EXE': '/cluster/miniconda3/bin/conda',
     'SLURM_STEP_NUM_NODES': '1',
     'SLURM_JOBID': '1612149',
     'SRUN_DEBUG': '3',
     'FTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'ftp_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_NTASKS': '24',
     'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
     'SLURM_STEP_ID': '0',
     'TMUX': '/tmp/tmux-67339/default,20557,4',
     '_CE_CONDA': '',
     'CONDA_PREFIX_1': '/cluster/miniconda3',
     'SLURM_STEP_LAUNCHER_PORT': '44912',
     'SLURM_TASKS_PER_NODE': '24',
     'MAIL': '/var/spool/mail/ye53nis',
     'PATH': '/home/ye53nis/.conda/envs/tf/bin:/home/lex/Programme/miniconda3/envs/tf-nightly-lab/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
     'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
     'SLURM_JOB_ID': '1612149',
     'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tf',
     'SLURM_JOB_USER': 'ye53nis',
     'SLURM_STEPID': '0',
     'PWD': '/',
     'SLURM_SRUN_COMM_HOST': '192.168.192.5',
     'LANG': 'en_US.UTF-8',
     'SLURM_PTY_WIN_COL': '236',
     'SLURM_UMASK': '0022',
     'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
     'SLURM_JOB_UID': '67339',
     'LOADEDMODULES': '',
     'SLURM_NODEID': '0',
     'TMUX_PANE': '%4',
     'SLURM_SUBMIT_DIR': '/',
     'SLURM_TASK_PID': '214733',
     'SLURM_NPROCS': '24',
     'SLURM_CPUS_ON_NODE': '24',
     'SLURM_DISTRIBUTION': 'block',
     'HTTPS_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'https_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_PROCID': '0',
     'HISTCONTROL': 'ignoredups',
     '_CE_M': '',
     'SLURM_JOB_NODELIST': 'node305',
     'SLURM_PTY_PORT': '44825',
     'HOME': '/home/ye53nis',
     'SHLVL': '3',
     'SLURM_LOCALID': '0',
     'SLURM_JOB_GID': '13280',
     'SLURM_JOB_CPUS_PER_NODE': '24',
     'SLURM_CLUSTER_NAME': 'hpc',
     'no_proxy': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001',
     'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
     'SLURM_SUBMIT_HOST': 'login01',
     'HTTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_JOB_PARTITION': 's_standard',
     'MATHEMATICA_HOME': '/cluster/apps/mathematica/11.3',
     'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
     'LOGNAME': 'ye53nis',
     'SLURM_STEP_NUM_TASKS': '24',
     'QTLIB': '/usr/lib64/qt-3.3/lib',
     'SLURM_JOB_ACCOUNT': 'iaob',
     'SLURM_JOB_NUM_NODES': '1',
     'MODULESHOME': '/usr/share/Modules',
     'CONDA_DEFAULT_ENV': 'tf',
     'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
     'SLURM_STEP_TASKS_PER_NODE': '24',
     'PORT': '8889',
     'SLURM_STEP_NODELIST': 'node305',
     'DISPLAY': ':0',
     'XDG_RUNTIME_DIR': '',
     'XAUTHORITY': '/home/lex/.Xauthority',
     'BASH_FUNC_module()': '() {  eval `/usr/bin/modulecmd bash $*`\n}',
     '_': '/home/ye53nis/.conda/envs/tf/bin/jupyter',
     'PYDEVD_USE_FRAME_EVAL': 'NO',
     'JPY_PARENT_PID': '214856',
     'CLICOLOR': '1',
     'PAGER': 'cat',
     'GIT_PAGER': 'cat',
     'MPLBACKEND': 'module://matplotlib_inline.backend_inline'}
    

2.5.2 Setup: Jupyter 2 on HPC compute node 2

  1. Setup tmux (#+CALL: setup-tmux[:session remote]) - done already above
  2. Request compute node via tmux
    cd /
    srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
    
    (base) [ye53nis@login01 /]$ srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2
    000 --pty bash
    (base) [ye53nis@node169 /]$
    
  3. Branch out git branch exp-220120-correlate-ptu from main (done via magit) and make sure ou are on the correct branch
    cd /beegfs/ye53nis/drmed-git
    git checkout exp-220120-correlate-ptu
    
    (base) [ye53nis@node145 drmed-git]$ git checkout exp-220120-correlate-ptu
    M       src/nanosimpy
    Already on 'exp-220120-correlate-ptu'
    (base) [ye53nis@node145 drmed-git]$
    
  4. Make directory for experiment - already done above
    ls data/exp-220120-correlate-ptu
    
    (base) [ye53nis@node145 drmed-git]$ ls data/exp-220120-correlate-ptu
    jupyter
    
  5. Set output folder using the following org-mode variable - already done above
  6. Load conda environment, and start jupyter
    conda activate tf
    export PORT=9997
    export XDG_RUNTIME_DIR=''
    export XDG_RUNTIME_DIR=""
    jupyter lab --no-browser --port=$PORT
    
    (tf) [ye53nis@node169 /]$ jupyter lab --no-browser --port=$PORT
    [I 2022-02-08 15:21:03.957 ServerApp] jupyterlab | extension was successfully linked.
    [I 2022-02-08 15:21:04.593 ServerApp] nbclassic | extension was successfully linked.
    [I 2022-02-08 15:21:04.659 LabApp] JupyterLab extension loaded from /home/ye53nis/.conda/envs/tf/lib/
    python3.9/site-packages/jupyterlab
    [I 2022-02-08 15:21:04.659 LabApp] JupyterLab application directory is /home/ye53nis/.conda/envs/tf/s
    hare/jupyter/lab
    [I 2022-02-08 15:21:04.670 ServerApp] jupyterlab | extension was successfully loaded.
    [I 2022-02-08 15:21:04.684 ServerApp] nbclassic | extension was successfully loaded.
    [I 2022-02-08 15:21:04.685 ServerApp] Serving notebooks from local directory: /
    [I 2022-02-08 15:21:04.685 ServerApp] Jupyter Server 1.4.1 is running at:
    [I 2022-02-08 15:21:04.685 ServerApp] http://localhost:9997/lab?token=9ac76fb054cea3bc3b076004eac0489
    60ff38213a8a87ed2
    [I 2022-02-08 15:21:04.685 ServerApp]  or http://127.0.0.1:9997/lab?token=9ac76fb054cea3bc3b076004eac
    048960ff38213a8a87ed2
    [I 2022-02-08 15:21:04.685 ServerApp] Use Control-C to stop this server and shut down all kernels (tw
    ice to skip confirmation).
    [C 2022-02-08 15:21:04.697 ServerApp]
    
        To access the server, open this file in a browser:
            file:///home/ye53nis/.local/share/jupyter/runtime/jpserver-319063-open.html
        Or copy and paste one of these URLs:
            http://localhost:9997/lab?token=9ac76fb054cea3bc3b076004eac048960ff38213a8a87ed2
         or http://127.0.0.1:9997/lab?token=9ac76fb054cea3bc3b076004eac048960ff38213a8a87ed2
    
  7. Create ssh tunnel (#+CALL: ssh-tunnel[:session org-tunnel2](port="9997", node="node169"))
                     
    sh-5.1$ sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:          
    ye53nis@node169's password:              
    Last login: Wed Feb 9 13:54:20 2022 from login01.ara
  8. I started a Python3 kernel using jupyter-server-list-kernels. Then I added the kernel ID to the :PROPERTIES: drawer of this (and following) subtrees.
    python3           03038b73-b2b5-49ce-a1dc-21afb6247d0f   a few seconds ago    starting   0
    
  9. Test: (#+CALL: jp-metadata(_long='True)) before update in Experiment 2a
    No of CPUs in system: 72
    No of CPUs the current process can use: 24
    load average: (35.8, 34.75, 28.33)
    os.uname():  posix.uname_result(sysname='Linux', nodename='node169', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
    PID of process: 320784
    RAM total: 199G, RAM used: 75G, RAM free: 75G
    the current directory: /
    My disk usage:
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/sda1             50G  3.8G   47G   8% /
    devtmpfs              94G     0   94G   0% /dev
    tmpfs                 94G  7.5M   94G   1% /dev/shm
    tmpfs                 94G  403M   94G   1% /run
    tmpfs                 94G     0   94G   0% /sys/fs/cgroup
    nfs03-ib:/pool/work  100T   79T   22T  79% /nfsdata
    nfs02-ib:/data01      88T   72T   16T  82% /data01
    nfs01-ib:/home        80T   67T   14T  84% /home
    nfs01-ib:/cluster    2.0T  468G  1.6T  23% /cluster
    /dev/sda6            169G  750M  168G   1% /local
    /dev/sda3            6.0G  414M  5.6G   7% /var
    /dev/sda5            2.0G   34M  2.0G   2% /tmp
    beegfs_nodev         524T  476T   49T  91% /beegfs
    tmpfs                 19G     0   19G   0% /run/user/67339# packages in environment at /home/ye53nis/.conda/envs/tf:
    #
    # Name                    Version                   Build  Channel
    _libgcc_mutex             0.1                        main
    _openmp_mutex             4.5                       1_gnu
    absl-py                   0.13.0                   pypi_0    pypi
    alembic                   1.4.1                    pypi_0    pypi
    anyio                     2.2.0            py39h06a4308_1
    argon2-cffi               20.1.0           py39h27cfd23_1
    asteval                   0.9.25                   pypi_0    pypi
    astunparse                1.6.3                    pypi_0    pypi
    async_generator           1.10               pyhd3eb1b0_0
    attrs                     21.2.0             pyhd3eb1b0_0
    babel                     2.9.1              pyhd3eb1b0_0
    backcall                  0.2.0              pyhd3eb1b0_0
    bleach                    3.3.1              pyhd3eb1b0_0
    brotlipy                  0.7.0           py39h27cfd23_1003
    ca-certificates           2021.7.5             h06a4308_1
    cachetools                4.2.2                    pypi_0    pypi
    certifi                   2021.5.30        py39h06a4308_0
    cffi                      1.14.6           py39h400218f_0
    chardet                   4.0.0           py39h06a4308_1003
    click                     8.0.1                    pypi_0    pypi
    cloudpickle               1.6.0                    pypi_0    pypi
    cryptography              3.4.7            py39hd23ed53_0
    cycler                    0.10.0                   pypi_0    pypi
    databricks-cli            0.14.3                   pypi_0    pypi
    decorator                 5.0.9              pyhd3eb1b0_0
    defusedxml                0.7.1              pyhd3eb1b0_0
    docker                    5.0.0                    pypi_0    pypi
    entrypoints               0.3              py39h06a4308_0
    fcsfiles                  2021.6.6                 pypi_0    pypi
    flask                     2.0.1                    pypi_0    pypi
    flatbuffers               1.12                     pypi_0    pypi
    future                    0.18.2                   pypi_0    pypi
    gast                      0.4.0                    pypi_0    pypi
    gitdb                     4.0.7                    pypi_0    pypi
    gitpython                 3.1.18                   pypi_0    pypi
    google-auth               1.34.0                   pypi_0    pypi
    google-auth-oauthlib      0.4.5                    pypi_0    pypi
    google-pasta              0.2.0                    pypi_0    pypi
    greenlet                  1.1.0                    pypi_0    pypi
    grpcio                    1.34.1                   pypi_0    pypi
    gunicorn                  20.1.0                   pypi_0    pypi
    h5py                      3.1.0                    pypi_0    pypi
    idna                      2.10               pyhd3eb1b0_0
    importlib-metadata        3.10.0           py39h06a4308_0
    importlib_metadata        3.10.0               hd3eb1b0_0
    ipykernel                 5.3.4            py39hb070fc8_0
    ipython                   7.22.0           py39hb070fc8_0
    ipython_genutils          0.2.0              pyhd3eb1b0_1
    itsdangerous              2.0.1                    pypi_0    pypi
    jedi                      0.17.2           py39h06a4308_1
    jinja2                    3.0.1              pyhd3eb1b0_0
    joblib                    1.0.1                    pypi_0    pypi
    json5                     0.9.6              pyhd3eb1b0_0
    jsonschema                3.2.0                      py_2
    jupyter-packaging         0.7.12             pyhd3eb1b0_0
    jupyter_client            6.1.12             pyhd3eb1b0_0
    jupyter_core              4.7.1            py39h06a4308_0
    jupyter_server            1.4.1            py39h06a4308_0
    jupyterlab                3.0.14             pyhd3eb1b0_1
    jupyterlab_pygments       0.1.2                      py_0
    jupyterlab_server         2.6.1              pyhd3eb1b0_0
    keras-nightly             2.5.0.dev2021032900          pypi_0    pypi
    keras-preprocessing       1.1.2                    pypi_0    pypi
    kiwisolver                1.3.1                    pypi_0    pypi
    ld_impl_linux-64          2.35.1               h7274673_9
    libffi                    3.3                  he6710b0_2
    libgcc-ng                 9.3.0               h5101ec6_17
    libgomp                   9.3.0               h5101ec6_17
    libsodium                 1.0.18               h7b6447c_0
    libstdcxx-ng              9.3.0               hd4cf53a_17
    lmfit                     1.0.2                    pypi_0    pypi
    mako                      1.1.4                    pypi_0    pypi
    markdown                  3.3.4                    pypi_0    pypi
    markupsafe                2.0.1            py39h27cfd23_0
    matplotlib                3.4.2                    pypi_0    pypi
    mistune                   0.8.4           py39h27cfd23_1000
    mlflow                    1.19.0                   pypi_0    pypi
    multipletau               0.3.3                    pypi_0    pypi
    nbclassic                 0.2.6              pyhd3eb1b0_0
    nbclient                  0.5.3              pyhd3eb1b0_0
    nbconvert                 6.1.0            py39h06a4308_0
    nbformat                  5.1.3              pyhd3eb1b0_0
    ncurses                   6.2                  he6710b0_1
    nest-asyncio              1.5.1              pyhd3eb1b0_0
    notebook                  6.4.0            py39h06a4308_0
    numpy                     1.19.5                   pypi_0    pypi
    oauthlib                  3.1.1                    pypi_0    pypi
    openssl                   1.1.1k               h27cfd23_0
    opt-einsum                3.3.0                    pypi_0    pypi
    packaging                 21.0               pyhd3eb1b0_0
    pandas                    1.3.1                    pypi_0    pypi
    pandocfilters             1.4.3            py39h06a4308_1
    parso                     0.7.0                      py_0
    pexpect                   4.8.0              pyhd3eb1b0_3
    pickleshare               0.7.5           pyhd3eb1b0_1003
    pillow                    8.3.1                    pypi_0    pypi
    pip                       21.1.3           py39h06a4308_0
    prometheus-flask-exporter 0.18.2                   pypi_0    pypi
    prometheus_client         0.11.0             pyhd3eb1b0_0
    prompt-toolkit            3.0.17             pyh06a4308_0
    protobuf                  3.17.3                   pypi_0    pypi
    ptyprocess                0.7.0              pyhd3eb1b0_2
    pyasn1                    0.4.8                    pypi_0    pypi
    pyasn1-modules            0.2.8                    pypi_0    pypi
    pycparser                 2.20                       py_2
    pygments                  2.9.0              pyhd3eb1b0_0
    pyopenssl                 20.0.1             pyhd3eb1b0_1
    pyparsing                 2.4.7              pyhd3eb1b0_0
    pyrsistent                0.18.0           py39h7f8727e_0
    pysocks                   1.7.1            py39h06a4308_0
    python                    3.9.5                h12debd9_4
    python-dateutil           2.8.2              pyhd3eb1b0_0
    python-editor             1.0.4                    pypi_0    pypi
    pytz                      2021.1             pyhd3eb1b0_0
    pyyaml                    5.4.1                    pypi_0    pypi
    pyzmq                     20.0.0           py39h2531618_1
    querystring-parser        1.2.4                    pypi_0    pypi
    readline                  8.1                  h27cfd23_0
    requests                  2.25.1             pyhd3eb1b0_0
    requests-oauthlib         1.3.0                    pypi_0    pypi
    rsa                       4.7.2                    pypi_0    pypi
    scikit-learn              0.24.2                   pypi_0    pypi
    scipy                     1.7.0                    pypi_0    pypi
    seaborn                   0.11.1                   pypi_0    pypi
    send2trash                1.5.0              pyhd3eb1b0_1
    setuptools                52.0.0           py39h06a4308_0
    six                       1.15.0                   pypi_0    pypi
    smmap                     4.0.0                    pypi_0    pypi
    sniffio                   1.2.0            py39h06a4308_1
    sqlalchemy                1.4.22                   pypi_0    pypi
    sqlite                    3.36.0               hc218d9a_0
    sqlparse                  0.4.1                    pypi_0    pypi
    tabulate                  0.8.9                    pypi_0    pypi
    tensorboard               2.5.0                    pypi_0    pypi
    tensorboard-data-server   0.6.1                    pypi_0    pypi
    tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
    tensorflow                2.5.0                    pypi_0    pypi
    tensorflow-estimator      2.5.0                    pypi_0    pypi
    termcolor                 1.1.0                    pypi_0    pypi
    terminado                 0.9.4            py39h06a4308_0
    testpath                  0.5.0              pyhd3eb1b0_0
    threadpoolctl             2.2.0                    pypi_0    pypi
    tifffile                  2021.7.30                pypi_0    pypi
    tk                        8.6.10               hbc83047_0
    tornado                   6.1              py39h27cfd23_0
    traitlets                 5.0.5              pyhd3eb1b0_0
    typing-extensions         3.7.4.3                  pypi_0    pypi
    tzdata                    2021a                h52ac0ba_0
    uncertainties             3.1.6                    pypi_0    pypi
    urllib3                   1.26.6             pyhd3eb1b0_1
    wcwidth                   0.2.5                      py_0
    webencodings              0.5.1            py39h06a4308_1
    websocket-client          1.1.0                    pypi_0    pypi
    werkzeug                  2.0.1                    pypi_0    pypi
    wheel                     0.36.2             pyhd3eb1b0_0
    wrapt                     1.12.1                   pypi_0    pypi
    xz                        5.2.5                h7b6447c_0
    zeromq                    4.3.4                h2531618_0
    zipp                      3.5.0              pyhd3eb1b0_0
    zlib                      1.2.11               h7b6447c_3
    
    Note: you may need to restart the kernel to use updated packages.
    {'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
     'SLURM_NODELIST': 'node169',
     'SLURM_JOB_NAME': 'bash',
     'XDG_SESSION_ID': '44301',
     'SLURMD_NODENAME': 'node169',
     'SLURM_TOPOLOGY_ADDR': 'node169',
     'SLURM_NTASKS_PER_NODE': '24',
     'HOSTNAME': 'login01',
     'SLURM_PRIO_PROCESS': '0',
     'SLURM_SRUN_COMM_PORT': '41490',
     'SHELL': '/bin/bash',
     'TERM': 'xterm-color',
     'SLURM_JOB_QOS': 'qstand',
     'SLURM_PTY_WIN_ROW': '27',
     'HISTSIZE': '1000',
     'TMPDIR': '/tmp',
     'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
     'SSH_CLIENT': '10.231.181.128 49370 22',
     'CONDA_SHLVL': '2',
     'CONDA_PROMPT_MODIFIER': '(tf) ',
     'QTDIR': '/usr/lib64/qt-3.3',
     'QTINC': '/usr/lib64/qt-3.3/include',
     'SSH_TTY': '/dev/pts/79',
     'NO_PROXY': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001',
     'QT_GRAPHICSSYSTEM_CHECKED': '1',
     'SLURM_NNODES': '1',
     'USER': 'ye53nis',
     'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
     'CONDA_EXE': '/cluster/miniconda3/bin/conda',
     'SLURM_STEP_NUM_NODES': '1',
     'SLURM_JOBID': '1608816',
     'SRUN_DEBUG': '3',
     'FTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'ftp_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_NTASKS': '24',
     'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
     'SLURM_STEP_ID': '0',
     'TMUX': '/tmp/tmux-67339/default,20557,9',
     '_CE_CONDA': '',
     'CONDA_PREFIX_1': '/cluster/miniconda3',
     'SLURM_STEP_LAUNCHER_PORT': '41490',
     'SLURM_TASKS_PER_NODE': '24',
     'MAIL': '/var/spool/mail/ye53nis',
     'PATH': '/home/ye53nis/.conda/envs/tf/bin:/home/lex/Programme/miniconda3/envs/tf/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
     'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
     'SLURM_JOB_ID': '1608816',
     'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tf',
     'SLURM_JOB_USER': 'ye53nis',
     'SLURM_STEPID': '0',
     'PWD': '/',
     'SLURM_SRUN_COMM_HOST': '192.168.192.5',
     'LANG': 'en_US.UTF-8',
     'SLURM_PTY_WIN_COL': '101',
     'SLURM_UMASK': '0022',
     'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
     'SLURM_JOB_UID': '67339',
     'LOADEDMODULES': '',
     'SLURM_NODEID': '0',
     'TMUX_PANE': '%9',
     'SLURM_SUBMIT_DIR': '/',
     'SLURM_TASK_PID': '318230',
     'SLURM_NPROCS': '24',
     'SLURM_CPUS_ON_NODE': '24',
     'SLURM_DISTRIBUTION': 'block',
     'HTTPS_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'https_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_PROCID': '0',
     'HISTCONTROL': 'ignoredups',
     '_CE_M': '',
     'SLURM_JOB_NODELIST': 'node169',
     'SLURM_PTY_PORT': '38396',
     'HOME': '/home/ye53nis',
     'SHLVL': '3',
     'SLURM_LOCALID': '0',
     'SLURM_JOB_GID': '13280',
     'SLURM_JOB_CPUS_PER_NODE': '24',
     'SLURM_CLUSTER_NAME': 'hpc',
     'no_proxy': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001',
     'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
     'SLURM_SUBMIT_HOST': 'login01',
     'HTTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_JOB_PARTITION': 's_standard',
     'MATHEMATICA_HOME': '/cluster/apps/mathematica/12.3',
     'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
     'LOGNAME': 'ye53nis',
     'SLURM_STEP_NUM_TASKS': '24',
     'QTLIB': '/usr/lib64/qt-3.3/lib',
     'SLURM_JOB_ACCOUNT': 'iaob',
     'SLURM_JOB_NUM_NODES': '1',
     'MODULESHOME': '/usr/share/Modules',
     'CONDA_DEFAULT_ENV': 'tf',
     'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
     'SLURM_STEP_TASKS_PER_NODE': '24',
     'PORT': '9997',
     'SLURM_STEP_NODELIST': 'node169',
     'DISPLAY': ':0',
     'XDG_RUNTIME_DIR': '',
     'XAUTHORITY': '/home/lex/.Xauthority',
     'BASH_FUNC_module()': '() {  eval `/usr/bin/modulecmd bash $*`\n}',
     '_': '/home/ye53nis/.conda/envs/tf/bin/jupyter',
     'JPY_PARENT_PID': '319063',
     'CLICOLOR': '1',
     'PAGER': 'cat',
     'GIT_PAGER': 'cat',
     'MPLBACKEND': 'module://ipykernel.pylab.backend_inline'}
    
  10. Test and save metadata after update in Experiment 2a
    No of CPUs in system: 72
    No of CPUs the current process can use: 24
    load average: (35.7, 25.74, 18.03)
    os.uname():  posix.uname_result(sysname='Linux', nodename='node169', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
    PID of process: 320784
    RAM total: 199G, RAM used: 75G, RAM free: 74G
    the current directory: /
    My disk usage:
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/sda1             50G  3.8G   47G   8% /
    devtmpfs              94G     0   94G   0% /dev
    tmpfs                 94G  7.5M   94G   1% /dev/shm
    tmpfs                 94G  403M   94G   1% /run
    tmpfs                 94G     0   94G   0% /sys/fs/cgroup
    nfs03-ib:/pool/work  100T   79T   22T  79% /nfsdata
    nfs02-ib:/data01      88T   72T   16T  82% /data01
    nfs01-ib:/home        80T   67T   14T  84% /home
    nfs01-ib:/cluster    2.0T  468G  1.6T  23% /cluster
    /dev/sda6            169G  750M  168G   1% /local
    /dev/sda3            6.0G  414M  5.6G   7% /var
    /dev/sda5            2.0G   34M  2.0G   2% /tmp
    beegfs_nodev         524T  476T   49T  91% /beegfs
    tmpfs                 19G     0   19G   0% /run/user/67339# packages in environment at /home/ye53nis/.conda/envs/tf:
    #
    # Name                    Version                   Build  Channel
    _libgcc_mutex             0.1                        main
    _openmp_mutex             4.5                       1_gnu
    absl-py                   1.0.0                    pypi_0    pypi
    alembic                   1.7.6                    pypi_0    pypi
    anyio                     2.2.0            py39h06a4308_1
    argon2-cffi               20.1.0           py39h27cfd23_1
    asteval                   0.9.26                   pypi_0    pypi
    astunparse                1.6.3                    pypi_0    pypi
    async_generator           1.10               pyhd3eb1b0_0
    attrs                     21.4.0             pyhd3eb1b0_0
    babel                     2.9.1              pyhd3eb1b0_0
    backcall                  0.2.0              pyhd3eb1b0_0
    bleach                    4.1.0              pyhd3eb1b0_0
    brotlipy                  0.7.0           py39h27cfd23_1003
    ca-certificates           2021.10.26           h06a4308_2
    cachetools                5.0.0                    pypi_0    pypi
    certifi                   2021.10.8        py39h06a4308_2
    cffi                      1.15.0           py39hd667e15_1
    charset-normalizer        2.0.4              pyhd3eb1b0_0
    click                     8.0.3                    pypi_0    pypi
    cloudpickle               2.0.0                    pypi_0    pypi
    cryptography              36.0.0           py39h9ce1e76_0
    cycler                    0.11.0                   pypi_0    pypi
    cython                    0.29.27                  pypi_0    pypi
    databricks-cli            0.16.4                   pypi_0    pypi
    debugpy                   1.5.1            py39h295c915_0
    decorator                 5.1.1              pyhd3eb1b0_0
    defusedxml                0.7.1              pyhd3eb1b0_0
    docker                    5.0.3                    pypi_0    pypi
    entrypoints               0.3              py39h06a4308_0
    fcsfiles                  2022.2.2                 pypi_0    pypi
    flask                     2.0.2                    pypi_0    pypi
    flatbuffers               2.0                      pypi_0    pypi
    fonttools                 4.29.1                   pypi_0    pypi
    future                    0.18.2                   pypi_0    pypi
    gast                      0.5.3                    pypi_0    pypi
    gitdb                     4.0.9                    pypi_0    pypi
    gitpython                 3.1.26                   pypi_0    pypi
    google-auth               2.6.0                    pypi_0    pypi
    google-auth-oauthlib      0.4.6                    pypi_0    pypi
    google-pasta              0.2.0                    pypi_0    pypi
    greenlet                  1.1.2                    pypi_0    pypi
    grpcio                    1.43.0                   pypi_0    pypi
    gunicorn                  20.1.0                   pypi_0    pypi
    h5py                      3.6.0                    pypi_0    pypi
    idna                      3.3                pyhd3eb1b0_0
    importlib-metadata        4.8.2            py39h06a4308_0
    importlib_metadata        4.8.2                hd3eb1b0_0
    ipykernel                 6.4.1            py39h06a4308_1
    ipython                   7.31.1           py39h06a4308_0
    ipython_genutils          0.2.0              pyhd3eb1b0_1
    itsdangerous              2.0.1                    pypi_0    pypi
    jedi                      0.18.1           py39h06a4308_1
    jinja2                    3.0.2              pyhd3eb1b0_0
    joblib                    1.1.0                    pypi_0    pypi
    json5                     0.9.6              pyhd3eb1b0_0
    jsonschema                3.2.0              pyhd3eb1b0_2
    jupyter_client            7.1.2              pyhd3eb1b0_0
    jupyter_core              4.9.1            py39h06a4308_0
    jupyter_server            1.4.1            py39h06a4308_0
    jupyterlab                3.2.1              pyhd3eb1b0_1
    jupyterlab_pygments       0.1.2                      py_0
    jupyterlab_server         2.10.2             pyhd3eb1b0_1
    keras                     2.8.0                    pypi_0    pypi
    keras-preprocessing       1.1.2                    pypi_0    pypi
    kiwisolver                1.3.2                    pypi_0    pypi
    ld_impl_linux-64          2.35.1               h7274673_9
    libclang                  13.0.0                   pypi_0    pypi
    libffi                    3.3                  he6710b0_2
    libgcc-ng                 9.3.0               h5101ec6_17
    libgomp                   9.3.0               h5101ec6_17
    libsodium                 1.0.18               h7b6447c_0
    libstdcxx-ng              9.3.0               hd4cf53a_17
    lmfit                     1.0.3                    pypi_0    pypi
    mako                      1.1.6                    pypi_0    pypi
    markdown                  3.3.6                    pypi_0    pypi
    markupsafe                2.0.1            py39h27cfd23_0
    matplotlib                3.5.1                    pypi_0    pypi
    matplotlib-inline         0.1.2              pyhd3eb1b0_2
    mistune                   0.8.4           py39h27cfd23_1000
    mlflow                    1.23.1                   pypi_0    pypi
    multipletau               0.3.3                    pypi_0    pypi
    nbclassic                 0.2.6              pyhd3eb1b0_0
    nbclient                  0.5.3              pyhd3eb1b0_0
    nbconvert                 6.3.0            py39h06a4308_0
    nbformat                  5.1.3              pyhd3eb1b0_0
    ncurses                   6.3                  h7f8727e_2
    nest-asyncio              1.5.1              pyhd3eb1b0_0
    notebook                  6.4.6            py39h06a4308_0
    numpy                     1.22.2                   pypi_0    pypi
    oauthlib                  3.2.0                    pypi_0    pypi
    openssl                   1.1.1m               h7f8727e_0
    opt-einsum                3.3.0                    pypi_0    pypi
    packaging                 21.3               pyhd3eb1b0_0
    pandas                    1.4.0                    pypi_0    pypi
    pandocfilters             1.5.0              pyhd3eb1b0_0
    parso                     0.8.3              pyhd3eb1b0_0
    pexpect                   4.8.0              pyhd3eb1b0_3
    pickleshare               0.7.5           pyhd3eb1b0_1003
    pillow                    9.0.1                    pypi_0    pypi
    pip                       21.2.4           py39h06a4308_0
    prometheus-flask-exporter 0.18.7                   pypi_0    pypi
    prometheus_client         0.13.1             pyhd3eb1b0_0
    prompt-toolkit            3.0.20             pyhd3eb1b0_0
    protobuf                  3.19.4                   pypi_0    pypi
    ptyprocess                0.7.0              pyhd3eb1b0_2
    pyasn1                    0.4.8                    pypi_0    pypi
    pyasn1-modules            0.2.8                    pypi_0    pypi
    pycparser                 2.21               pyhd3eb1b0_0
    pygments                  2.11.2             pyhd3eb1b0_0
    pyopenssl                 22.0.0             pyhd3eb1b0_0
    pyparsing                 3.0.4              pyhd3eb1b0_0
    pyrsistent                0.18.0           py39heee7806_0
    pysocks                   1.7.1            py39h06a4308_0
    python                    3.9.7                h12debd9_1
    python-dateutil           2.8.2              pyhd3eb1b0_0
    pytz                      2021.3             pyhd3eb1b0_0
    pyyaml                    6.0                      pypi_0    pypi
    pyzmq                     22.3.0           py39h295c915_2
    querystring-parser        1.2.4                    pypi_0    pypi
    readline                  8.1.2                h7f8727e_1
    requests                  2.27.1             pyhd3eb1b0_0
    requests-oauthlib         1.3.1                    pypi_0    pypi
    rsa                       4.8                      pypi_0    pypi
    scikit-learn              1.0.2                    pypi_0    pypi
    scipy                     1.8.0                    pypi_0    pypi
    seaborn                   0.11.2                   pypi_0    pypi
    send2trash                1.8.0              pyhd3eb1b0_1
    setuptools                58.0.4           py39h06a4308_0
    six                       1.16.0             pyhd3eb1b0_0
    smmap                     5.0.0                    pypi_0    pypi
    sniffio                   1.2.0            py39h06a4308_1
    sqlalchemy                1.4.31                   pypi_0    pypi
    sqlite                    3.37.2               hc218d9a_0
    sqlparse                  0.4.2                    pypi_0    pypi
    tabulate                  0.8.9                    pypi_0    pypi
    tensorboard               2.8.0                    pypi_0    pypi
    tensorboard-data-server   0.6.1                    pypi_0    pypi
    tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
    tensorflow                2.8.0                    pypi_0    pypi
    tensorflow-io-gcs-filesystem 0.24.0                   pypi_0    pypi
    termcolor                 1.1.0                    pypi_0    pypi
    terminado                 0.9.4            py39h06a4308_0
    testpath                  0.5.0              pyhd3eb1b0_0
    tf-estimator-nightly      2.8.0.dev2021122109          pypi_0    pypi
    threadpoolctl             3.1.0                    pypi_0    pypi
    tk                        8.6.11               h1ccaba5_0
    tornado                   6.1              py39h27cfd23_0
    traitlets                 5.1.1              pyhd3eb1b0_0
    typing-extensions         4.0.1                    pypi_0    pypi
    tzdata                    2021e                hda174b7_0
    uncertainties             3.1.6                    pypi_0    pypi
    urllib3                   1.26.8             pyhd3eb1b0_0
    wcwidth                   0.2.5              pyhd3eb1b0_0
    webencodings              0.5.1            py39h06a4308_1
    websocket-client          1.2.3                    pypi_0    pypi
    werkzeug                  2.0.3                    pypi_0    pypi
    wheel                     0.37.1             pyhd3eb1b0_0
    wrapt                     1.13.3                   pypi_0    pypi
    xz                        5.2.5                h7b6447c_0
    zeromq                    4.3.4                h2531618_0
    zipp                      3.7.0              pyhd3eb1b0_0
    zlib                      1.2.11               h7f8727e_4
    
    Note: you may need to restart the kernel to use updated packages.
    {'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
     'SLURM_NODELIST': 'node169',
     'SLURM_JOB_NAME': 'bash',
     'XDG_SESSION_ID': '44301',
     'SLURMD_NODENAME': 'node169',
     'SLURM_TOPOLOGY_ADDR': 'node169',
     'SLURM_NTASKS_PER_NODE': '24',
     'HOSTNAME': 'login01',
     'SLURM_PRIO_PROCESS': '0',
     'SLURM_SRUN_COMM_PORT': '41490',
     'SHELL': '/bin/bash',
     'TERM': 'xterm-color',
     'SLURM_JOB_QOS': 'qstand',
     'SLURM_PTY_WIN_ROW': '27',
     'HISTSIZE': '1000',
     'TMPDIR': '/tmp',
     'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
     'SSH_CLIENT': '10.231.181.128 49370 22',
     'CONDA_SHLVL': '2',
     'CONDA_PROMPT_MODIFIER': '(tf) ',
     'QTDIR': '/usr/lib64/qt-3.3',
     'QTINC': '/usr/lib64/qt-3.3/include',
     'SSH_TTY': '/dev/pts/79',
     'NO_PROXY': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001',
     'QT_GRAPHICSSYSTEM_CHECKED': '1',
     'SLURM_NNODES': '1',
     'USER': 'ye53nis',
     'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
     'CONDA_EXE': '/cluster/miniconda3/bin/conda',
     'SLURM_STEP_NUM_NODES': '1',
     'SLURM_JOBID': '1608816',
     'SRUN_DEBUG': '3',
     'FTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'ftp_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_NTASKS': '24',
     'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
     'SLURM_STEP_ID': '0',
     'TMUX': '/tmp/tmux-67339/default,20557,9',
     '_CE_CONDA': '',
     'CONDA_PREFIX_1': '/cluster/miniconda3',
     'SLURM_STEP_LAUNCHER_PORT': '41490',
     'SLURM_TASKS_PER_NODE': '24',
     'MAIL': '/var/spool/mail/ye53nis',
     'PATH': '/home/ye53nis/.conda/envs/tf/bin:/home/lex/Programme/miniconda3/envs/tf/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
     'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
     'SLURM_JOB_ID': '1608816',
     'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tf',
     'SLURM_JOB_USER': 'ye53nis',
     'SLURM_STEPID': '0',
     'PWD': '/',
     'SLURM_SRUN_COMM_HOST': '192.168.192.5',
     'LANG': 'en_US.UTF-8',
     'SLURM_PTY_WIN_COL': '101',
     'SLURM_UMASK': '0022',
     'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
     'SLURM_JOB_UID': '67339',
     'LOADEDMODULES': '',
     'SLURM_NODEID': '0',
     'TMUX_PANE': '%9',
     'SLURM_SUBMIT_DIR': '/',
     'SLURM_TASK_PID': '318230',
     'SLURM_NPROCS': '24',
     'SLURM_CPUS_ON_NODE': '24',
     'SLURM_DISTRIBUTION': 'block',
     'HTTPS_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'https_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_PROCID': '0',
     'HISTCONTROL': 'ignoredups',
     '_CE_M': '',
     'SLURM_JOB_NODELIST': 'node169',
     'SLURM_PTY_PORT': '38396',
     'HOME': '/home/ye53nis',
     'SHLVL': '3',
     'SLURM_LOCALID': '0',
     'SLURM_JOB_GID': '13280',
     'SLURM_JOB_CPUS_PER_NODE': '24',
     'SLURM_CLUSTER_NAME': 'hpc',
     'no_proxy': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001',
     'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
     'SLURM_SUBMIT_HOST': 'login01',
     'HTTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_JOB_PARTITION': 's_standard',
     'MATHEMATICA_HOME': '/cluster/apps/mathematica/12.3',
     'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
     'LOGNAME': 'ye53nis',
     'SLURM_STEP_NUM_TASKS': '24',
     'QTLIB': '/usr/lib64/qt-3.3/lib',
     'SLURM_JOB_ACCOUNT': 'iaob',
     'SLURM_JOB_NUM_NODES': '1',
     'MODULESHOME': '/usr/share/Modules',
     'CONDA_DEFAULT_ENV': 'tf',
     'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
     'SLURM_STEP_TASKS_PER_NODE': '24',
     'PORT': '9997',
     'SLURM_STEP_NODELIST': 'node169',
     'DISPLAY': ':0',
     'XDG_RUNTIME_DIR': '',
     'XAUTHORITY': '/home/lex/.Xauthority',
     'BASH_FUNC_module()': '() {  eval `/usr/bin/modulecmd bash $*`\n}',
     '_': '/home/ye53nis/.conda/envs/tf/bin/jupyter',
     'JPY_PARENT_PID': '319063',
     'CLICOLOR': '1',
     'PAGER': 'cat',
     'GIT_PAGER': 'cat',
     'MPLBACKEND': 'module://ipykernel.pylab.backend_inline'}
    

2.5.3 Setup: Jupyter 3 on local computer

  1. on our local machine we don't need tmux. A simple sh command is enough. So let's start the conda environment in the sh session local and start jupterlab there.
    conda activate tf
    jupyter lab --no-browser --port=8888
    
    sh-5.1$ [I 2022-01-25 13:54:15.779 ServerApp] jupyterlab | extension was successfully linked.
    [I 2022-01-25 13:54:16.426 ServerApp] nbclassic | extension was successfully linked.
    [I 2022-01-25 13:54:16.509 LabApp] JupyterLab extension loaded from /home/lex/Programme/miniconda3/envs/tf/lib/python3.9/site-packages/jupyterlab
    [I 2022-01-25 13:54:16.509 LabApp] JupyterLab application directory is /home/lex/Programme/miniconda3/envs/tf/share/jupyter/lab
    [I 2022-01-25 13:54:16.517 ServerApp] jupyterlab | extension was successfully loaded.
    [I 2022-01-25 13:54:16.529 ServerApp] nbclassic | extension was successfully loaded.
    [I 2022-01-25 13:54:16.530 ServerApp] Serving notebooks from local directory: /home/lex/Programme/drmed-git
    [I 2022-01-25 13:54:16.530 ServerApp] Jupyter Server 1.4.1 is running at:
    [I 2022-01-25 13:54:16.530 ServerApp] http://localhost:8888/lab?token=a45ac98a52ac86cdccd87547a734345bdc338322def30d98
    [I 2022-01-25 13:54:16.530 ServerApp]  or http://127.0.0.1:8888/lab?token=a45ac98a52ac86cdccd87547a734345bdc338322def30d98
    [I 2022-01-25 13:54:16.530 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
    [C 2022-01-25 13:54:16.540 ServerApp]
    
        To access the server, open this file in a browser:
            file:///home/lex/.local/share/jupyter/runtime/jpserver-62432-open.html
        Or copy and paste one of these URLs:
            http://localhost:8888/lab?token=a45ac98a52ac86cdccd87547a734345bdc338322def30d98
         or http://127.0.0.1:8888/lab?token=a45ac98a52ac86cdccd87547a734345bdc338322def30d98
    
  2. I started a Python3 kernel using jupyter-server-list-kernels. Then I added the kernel ID to the :PROPERTIES: drawer of this (and following) subtrees.
    python3           03038b73-b2b5-49ce-a1dc-21afb6247d0f   a few seconds ago    starting   0
    
  3. Test: (#+CALL: jp-metadata(_long='True))
    No of CPUs in system: 4
    No of CPUs the current process can use: 4
    load average: (3.83, 4.72, 5.29)
    os.uname():  posix.uname_result(sysname='Linux', nodename='Topialex', release='5.13.19-2-MANJARO', version='#1 SMP PREEMPT Sun Sep 19 21:31:53 UTC 2021', machine='x86_64')
    PID of process: 321569
    RAM total: 16Gi, RAM used: 3,2Gi, RAM free: 9,1Gi
    the current directory: /home/lex/Programme/drmed-git
    My disk usage:
    Filesystem      Size  Used Avail Use% Mounted on
    dev             3,9G     0  3,9G   0% /dev
    run             3,9G  1,5M  3,9G   1% /run
    /dev/sda2       167G  130G   29G  82% /
    tmpfs           3,9G   96M  3,8G   3% /dev/shm
    tmpfs           3,9G   18M  3,9G   1% /tmp
    /dev/sda1       300M  264K  300M   1% /boot/efi
    tmpfs           784M  100K  784M   1% /run/user/1000# packages in environment at /home/lex/Programme/miniconda3/envs/tf:
    #
    # Name                    Version                   Build  Channel
    _libgcc_mutex             0.1                        main
    _openmp_mutex             4.5                       1_gnu
    absl-py                   1.0.0                    pypi_0    pypi
    alembic                   1.4.1                    pypi_0    pypi
    anyio                     2.2.0            py39h06a4308_1
    argon2-cffi               20.1.0           py39h27cfd23_1
    asteval                   0.9.25                   pypi_0    pypi
    astroid                   2.9.2                    pypi_0    pypi
    astunparse                1.6.3                    pypi_0    pypi
    async_generator           1.10               pyhd3eb1b0_0
    attrs                     21.2.0             pyhd3eb1b0_0
    babel                     2.9.1              pyhd3eb1b0_0
    backcall                  0.2.0              pyhd3eb1b0_0
    bleach                    4.0.0              pyhd3eb1b0_0
    brotlipy                  0.7.0           py39h27cfd23_1003
    ca-certificates           2021.10.26           h06a4308_2
    cachetools                4.2.4                    pypi_0    pypi
    certifi                   2021.10.8        py39h06a4308_0
    cffi                      1.14.6           py39h400218f_0
    charset-normalizer        2.0.4              pyhd3eb1b0_0
    click                     8.0.3                    pypi_0    pypi
    cloudpickle               2.0.0                    pypi_0    pypi
    cryptography              36.0.0           py39h9ce1e76_0
    cycler                    0.11.0                   pypi_0    pypi
    cython                    0.29.26                  pypi_0    pypi
    databricks-cli            0.16.2                   pypi_0    pypi
    debugpy                   1.5.1            py39h295c915_0
    decorator                 5.1.0              pyhd3eb1b0_0
    defusedxml                0.7.1              pyhd3eb1b0_0
    docker                    5.0.3                    pypi_0    pypi
    entrypoints               0.3              py39h06a4308_0
    fcsfiles                  2021.6.6                 pypi_0    pypi
    flake8                    4.0.1                    pypi_0    pypi
    flask                     2.0.2                    pypi_0    pypi
    flatbuffers               2.0                      pypi_0    pypi
    focuspoint                0.1                      pypi_0    pypi
    fonttools                 4.28.5                   pypi_0    pypi
    future                    0.18.2                   pypi_0    pypi
    gast                      0.4.0                    pypi_0    pypi
    gitdb                     4.0.9                    pypi_0    pypi
    gitpython                 3.1.24                   pypi_0    pypi
    google-auth               2.3.3                    pypi_0    pypi
    google-auth-oauthlib      0.4.6                    pypi_0    pypi
    google-pasta              0.2.0                    pypi_0    pypi
    greenlet                  1.1.2                    pypi_0    pypi
    grpcio                    1.43.0                   pypi_0    pypi
    gunicorn                  20.1.0                   pypi_0    pypi
    h5py                      3.6.0                    pypi_0    pypi
    idna                      3.3                pyhd3eb1b0_0
    importlib-metadata        4.8.2            py39h06a4308_0
    importlib_metadata        4.8.2                hd3eb1b0_0
    ipykernel                 6.4.1            py39h06a4308_1
    ipython                   7.29.0           py39hb070fc8_0
    ipython_genutils          0.2.0              pyhd3eb1b0_1
    isort                     5.10.1                   pypi_0    pypi
    itsdangerous              2.0.1                    pypi_0    pypi
    jedi                      0.18.0           py39h06a4308_1
    jinja2                    3.0.2              pyhd3eb1b0_0
    joblib                    1.1.0                    pypi_0    pypi
    json5                     0.9.6              pyhd3eb1b0_0
    jsonschema                3.2.0              pyhd3eb1b0_2
    jupyter_client            7.1.0              pyhd3eb1b0_0
    jupyter_core              4.9.1            py39h06a4308_0
    jupyter_server            1.4.1            py39h06a4308_0
    jupyterlab                3.2.1              pyhd3eb1b0_1
    jupyterlab_pygments       0.1.2                      py_0
    jupyterlab_server         2.8.2              pyhd3eb1b0_0
    keras                     2.7.0                    pypi_0    pypi
    keras-preprocessing       1.1.2                    pypi_0    pypi
    kiwisolver                1.3.2                    pypi_0    pypi
    lazy-object-proxy         1.7.1                    pypi_0    pypi
    ld_impl_linux-64          2.35.1               h7274673_9
    libclang                  12.0.0                   pypi_0    pypi
    libffi                    3.3                  he6710b0_2
    libgcc-ng                 9.3.0               h5101ec6_17
    libgomp                   9.3.0               h5101ec6_17
    libsodium                 1.0.18               h7b6447c_0
    libstdcxx-ng              9.3.0               hd4cf53a_17
    lmfit                     1.0.3                    pypi_0    pypi
    mako                      1.1.6                    pypi_0    pypi
    markdown                  3.3.6                    pypi_0    pypi
    markupsafe                2.0.1            py39h27cfd23_0
    matplotlib                3.5.1                    pypi_0    pypi
    matplotlib-inline         0.1.2              pyhd3eb1b0_2
    mccabe                    0.6.1                    pypi_0    pypi
    mistune                   0.8.4           py39h27cfd23_1000
    mlflow                    1.22.0                   pypi_0    pypi
    multipletau               0.3.3                    pypi_0    pypi
    mypy                      0.930                    pypi_0    pypi
    mypy-extensions           0.4.3                    pypi_0    pypi
    nbclassic                 0.2.6              pyhd3eb1b0_0
    nbclient                  0.5.3              pyhd3eb1b0_0
    nbconvert                 6.1.0            py39h06a4308_0
    nbformat                  5.1.3              pyhd3eb1b0_0
    ncurses                   6.3                  h7f8727e_2
    nest-asyncio              1.5.1              pyhd3eb1b0_0
    nodeenv                   1.6.0                    pypi_0    pypi
    notebook                  6.4.6            py39h06a4308_0
    numpy                     1.21.5                   pypi_0    pypi
    oauthlib                  3.1.1                    pypi_0    pypi
    openssl                   1.1.1l               h7f8727e_0
    opt-einsum                3.3.0                    pypi_0    pypi
    packaging                 21.3               pyhd3eb1b0_0
    pandas                    1.3.5                    pypi_0    pypi
    pandocfilters             1.4.3            py39h06a4308_1
    parso                     0.8.2              pyhd3eb1b0_0
    pexpect                   4.8.0              pyhd3eb1b0_3
    pickleshare               0.7.5           pyhd3eb1b0_1003
    pillow                    8.4.0                    pypi_0    pypi
    pip                       21.2.4           py39h06a4308_0
    platformdirs              2.4.1                    pypi_0    pypi
    prometheus-flask-exporter 0.18.7                   pypi_0    pypi
    prometheus_client         0.12.0             pyhd3eb1b0_0
    prompt-toolkit            3.0.20             pyhd3eb1b0_0
    protobuf                  3.19.1                   pypi_0    pypi
    ptyprocess                0.7.0              pyhd3eb1b0_2
    pyasn1                    0.4.8                    pypi_0    pypi
    pyasn1-modules            0.2.8                    pypi_0    pypi
    pycodestyle               2.8.0                    pypi_0    pypi
    pycparser                 2.21               pyhd3eb1b0_0
    pyflakes                  2.4.0                    pypi_0    pypi
    pygments                  2.10.0             pyhd3eb1b0_0
    pylint                    2.12.2                   pypi_0    pypi
    pyopenssl                 21.0.0             pyhd3eb1b0_1
    pyparsing                 3.0.4              pyhd3eb1b0_0
    pyright                   0.0.13                   pypi_0    pypi
    pyrsistent                0.18.0           py39heee7806_0
    pysocks                   1.7.1            py39h06a4308_0
    python                    3.9.7                h12debd9_1
    python-dateutil           2.8.2              pyhd3eb1b0_0
    python-editor             1.0.4                    pypi_0    pypi
    pytz                      2021.3             pyhd3eb1b0_0
    pyyaml                    6.0                      pypi_0    pypi
    pyzmq                     22.3.0           py39h295c915_2
    querystring-parser        1.2.4                    pypi_0    pypi
    readline                  8.1                  h27cfd23_0
    requests                  2.26.0             pyhd3eb1b0_0
    requests-oauthlib         1.3.0                    pypi_0    pypi
    rsa                       4.8                      pypi_0    pypi
    scikit-learn              1.0.2                    pypi_0    pypi
    scipy                     1.7.3                    pypi_0    pypi
    seaborn                   0.11.2                   pypi_0    pypi
    send2trash                1.8.0              pyhd3eb1b0_1
    setuptools                58.0.4           py39h06a4308_0
    six                       1.16.0             pyhd3eb1b0_0
    smmap                     5.0.0                    pypi_0    pypi
    sniffio                   1.2.0            py39h06a4308_1
    sqlalchemy                1.4.29                   pypi_0    pypi
    sqlite                    3.37.0               hc218d9a_0
    sqlparse                  0.4.2                    pypi_0    pypi
    tabulate                  0.8.9                    pypi_0    pypi
    tensorboard               2.7.0                    pypi_0    pypi
    tensorboard-data-server   0.6.1                    pypi_0    pypi
    tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
    tensorflow                2.7.0                    pypi_0    pypi
    tensorflow-estimator      2.7.0                    pypi_0    pypi
    tensorflow-io-gcs-filesystem 0.23.1                   pypi_0    pypi
    termcolor                 1.1.0                    pypi_0    pypi
    terminado                 0.9.4            py39h06a4308_0
    testpath                  0.5.0              pyhd3eb1b0_0
    threadpoolctl             3.0.0                    pypi_0    pypi
    tk                        8.6.11               h1ccaba5_0
    toml                      0.10.2                   pypi_0    pypi
    tomli                     2.0.0                    pypi_0    pypi
    tornado                   6.1              py39h27cfd23_0
    traitlets                 5.1.1              pyhd3eb1b0_0
    typing-extensions         4.0.1                    pypi_0    pypi
    tzdata                    2021e                hda174b7_0
    uncertainties             3.1.6                    pypi_0    pypi
    urllib3                   1.26.7             pyhd3eb1b0_0
    wcwidth                   0.2.5              pyhd3eb1b0_0
    webencodings              0.5.1            py39h06a4308_1
    websocket-client          1.2.3                    pypi_0    pypi
    werkzeug                  2.0.2                    pypi_0    pypi
    wheel                     0.37.0             pyhd3eb1b0_1
    wrapt                     1.13.3                   pypi_0    pypi
    xz                        5.2.5                h7b6447c_0
    zeromq                    4.3.4                h2531618_0
    zipp                      3.6.0              pyhd3eb1b0_0
    zlib                      1.2.11               h7f8727e_4
    
    Note: you may need to restart the kernel to use updated packages.
    {'SHELL': '/bin/bash',
     'SESSION_MANAGER': 'local/Topialex:@/tmp/.ICE-unix/938,unix/Topialex:/tmp/.ICE-unix/938',
     'XDG_CONFIG_DIRS': '/home/lex/.config/kdedefaults:/etc/xdg',
     'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session1',
     'CONDA_EXE': '/home/lex/Programme/miniconda3/bin/conda',
     '_CE_M': '',
     'LANGUAGE': 'en_GB',
     'TERMCAP': '',
     'LC_ADDRESS': 'de_DE.UTF-8',
     'LC_NAME': 'de_DE.UTF-8',
     'INSIDE_EMACS': '27.2,comint',
     'DESKTOP_SESSION': 'plasma',
     'LC_MONETARY': 'de_DE.UTF-8',
     'GTK_RC_FILES': '/etc/gtk/gtkrc:/home/lex/.gtkrc:/home/lex/.config/gtkrc',
     'XCURSOR_SIZE': '24',
     'GTK_MODULES': 'canberra-gtk-module',
     'XDG_SEAT': 'seat0',
     'PWD': '/home/lex/Programme/drmed-git',
     'LOGNAME': 'lex',
     'XDG_SESSION_DESKTOP': 'KDE',
     'XDG_SESSION_TYPE': 'x11',
     'CONDA_PREFIX': '/home/lex/Programme/miniconda3/envs/tf',
     'DSSI_PATH': '/home/lex/.dssi:/usr/lib/dssi:/usr/local/lib/dssi',
     'SYSTEMD_EXEC_PID': '819',
     'XAUTHORITY': '/home/lex/.Xauthority',
     'MOTD_SHOWN': 'pam',
     'GTK2_RC_FILES': '/etc/gtk-2.0/gtkrc:/home/lex/.gtkrc-2.0:/home/lex/.config/gtkrc-2.0',
     'HOME': '/home/lex',
     'LANG': 'de_DE.UTF-8',
     'LC_PAPER': 'de_DE.UTF-8',
     'VST_PATH': '/home/lex/.vst:/usr/lib/vst:/usr/local/lib/vst',
     'XDG_CURRENT_DESKTOP': 'KDE',
     'COLUMNS': '80',
     'CONDA_PROMPT_MODIFIER': '',
     'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0',
     'KDE_SESSION_UID': '1000',
     'XDG_SESSION_CLASS': 'user',
     'LC_IDENTIFICATION': 'de_DE.UTF-8',
     'TERM': 'xterm-color',
     '_CE_CONDA': '',
     'USER': 'lex',
     'CONDA_SHLVL': '1',
     'KDE_SESSION_VERSION': '5',
     'PAM_KWALLET5_LOGIN': '/run/user/1000/kwallet5.socket',
     'DISPLAY': ':0',
     'SHLVL': '2',
     'LC_TELEPHONE': 'de_DE.UTF-8',
     'LC_MEASUREMENT': 'de_DE.UTF-8',
     'XDG_VTNR': '1',
     'XDG_SESSION_ID': '2',
     'QT_LINUX_ACCESSIBILITY_ALWAYS_ON': '1',
     'CONDA_PYTHON_EXE': '/home/lex/Programme/miniconda3/bin/python',
     'MOZ_PLUGIN_PATH': '/usr/lib/mozilla/plugins',
     'XDG_RUNTIME_DIR': '/run/user/1000',
     'CONDA_DEFAULT_ENV': 'tf',
     'LC_TIME': 'de_DE.UTF-8',
     'QT_AUTO_SCREEN_SCALE_FACTOR': '0',
     'XCURSOR_THEME': 'breeze_cursors',
     'XDG_DATA_DIRS': '/home/lex/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share:/var/lib/snapd/desktop',
     'KDE_FULL_SESSION': 'true',
     'BROWSER': 'vivaldi-stable',
     'PATH': '/home/lex/Programme/miniconda3/envs/tf/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin',
     'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus',
     'LV2_PATH': '/home/lex/.lv2:/usr/lib/lv2:/usr/local/lib/lv2',
     'KDE_APPLICATIONS_AS_SCOPE': '1',
     'MAIL': '/var/spool/mail/lex',
     'LC_NUMERIC': 'de_DE.UTF-8',
     'LADSPA_PATH': '/home/lex/.ladspa:/usr/lib/ladspa:/usr/local/lib/ladspa',
     'CADENCE_AUTO_STARTED': 'true',
     '_': '/home/lex/Programme/miniconda3/envs/tf/bin/jupyter',
     'PYDEVD_USE_FRAME_EVAL': 'NO',
     'JPY_PARENT_PID': '320867',
     'CLICOLOR': '1',
     'PAGER': 'cat',
     'GIT_PAGER': 'cat',
     'MPLBACKEND': 'module://matplotlib_inline.backend_inline'}
    

2.5.4 Experiment 1a: Correlate all_clean_ptu on Jupyter 1

  1. Make sure we are in the correct directory, also do a git log -3 to document latest git commits
    %cd /beegfs/ye53nis/drmed-git
    
    !git log -3
    
    /beegfs/ye53nis/drmed-git
    commit e2037847eadb43f0eaaeeb48cda3cc48141e24f2
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Wed Jan 19 20:10:28 2022 +0100
    
        Fix tt_key again
    
    commit 7c750e0c8ed328033d1e55b7331f5d8d8dedb4b5
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Wed Jan 19 20:07:41 2022 +0100
    
        Fix tt_key
    
    commit 6a23a927a62b1c05e3e6e438c9af4d6dc7791b48
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Wed Jan 19 20:01:15 2022 +0100
    
        Fix tttr2xfcs keyError when using standard name
    
  2. Create directory where we want to save our correlations in
    %mkdir /beegfs/ye53nis/saves/2022-01-20_correlate-all-clean-ptu/
    
  3. Load all needed modules
    import mlflow
    import logging
    import os
    import sys
    import matplotlib.pyplot as plt
    import numpy as np
    import tensorflow as tf
    from pathlib import Path
    from pprint import pprint
    
    FLUOTRACIFY_PATH = '/beegfs/ye53nis/drmed-git/src/'
    sys.path.append(FLUOTRACIFY_PATH)
    from fluotracify.applications import corr_fit_object as cfo
    from fluotracify.training import build_model as bm
    
    logging.basicConfig(filename='data/exp-220120-correlate-ptu/2022-01-20_correlate-all-clean-ptu.log',
                        filemode='w', format='%(asctime)s - %(message)s',
                        force=True,
                        level=logging.DEBUG)
    
    
  4. Define variables and prepare model
    class ParameterClass():
        """Stores parameters for correlation """
        def __init__(self):
            # Where the data is stored.
            self.data = []
            self.objectRef = []
            self.subObjectRef = []
            self.colors = ['blue', 'green', 'red', 'cyan', 'magenta',
                           'yellow', 'black']
            self.numOfLoaded = 0
            # very fast from Ncasc ~ 14 onwards
            self.NcascStart = 0
            self.NcascEnd = 30  # 25
            self.Nsub = 6  # 6
            self.photonLifetimeBin = 10  # used for photon decay
            self.photonCountBin = 1  # used for time series
    
    path = "../data/Pablo_structured_experiment/all_clean_ptu"
    logged_model = 'file:///beegfs/ye53nis/drmed-git/data/mlruns/8/00f2635d9fa2463c9a066722163405be/artifacts/model'
    weight = 0
    output_path = '/beegfs/ye53nis/saves/2022-01-20_correlate-all-clean-ptu/'
    
    par_obj = ParameterClass()
    
    loaded_model = mlflow.keras.load_model(logged_model, compile=False)
    loaded_model.compile(loss=bm.binary_ce_dice_loss(),
                         optimizer=tf.keras.optimizers.Adam(),
                         metrics = bm.unet_metrics([0.1, 0.3, 0.5, 0.7, 0.9]))
    bm.prepare_model(loaded_model)
    
    
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    
  5. Run experiment: Correlate all_clean_ptu on Jupyter 1
    path = Path(path)
    files = [path / f for f in os.listdir(path) if f.endswith('.ptu')]
    
    if len(files) == 0:
        raise FileNotFoundError('The path provided does not include any'
                                ' .ptu files.')
    for myfile in files:
        ptufile = cfo.PicoObject(myfile, par_obj)
        ptufile.predictTimeSeries(model=loaded_model,
                                  scaler='minmax')
        ptufile.correctTCSPC(method='weights',
                             weight=weight)
        shift_name = f'shift_{ptufile.name}'
        ptufile.getTimeSeries(timeseries_name=shift_name)
        ptufile.getPhotonCountingStats(name=shift_name)
        ptufile.predictTimeSeries(model=loaded_model,
                                  scaler='minmax',
                                  name=shift_name)
        ptufile.correctTCSPC(method='delete_and_shift',
                             timeseries_name=shift_name)
        for key in list(ptufile.trueTimeArr.keys()):
            if "_FORWEIGHTS" in key:
                ptufile.get_autocorrelation(method='tttr2xfcs_with_weights',
                                            name=key)
            else:
                ptufile.get_autocorrelation(method='tttr2xfcs', name=key)
    
        for m in ['multipletau', 'tttr2xfcs', 'tttr2xfcs_with_weights']:
            if m in list(ptufile.autoNorm.keys()):
                for key in list(ptufile.autoNorm[m].keys()):
                    ptufile.save_autocorrelation(name=key, method=m,
                                                 output_path=output_path)
    
    
    c4358abc-9c3b-458a-b64c-9d5e34fa28b0
    
  6. According to log, the first run ended with an Error
    [I 2022-01-22 21:32:42.402 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), keep random ports
    

    Only 229 traces were analyzed.

  7. Analyze using a different weight
    %cd /beegfs/ye53nis/drmed-git
    
    !git log -3
    
    /beegfs/ye53nis/drmed-git
    commit 9e735ecd10d0e5e5eae591f8cc40201ae6144b6a
    Author: Alex Seltmann <seltmann@posteo.de>
    Date:   Mon Jan 24 20:34:54 2022 +0100
    
        add correlations 2
    
    commit c0b4bc8494409a6b83a623977f07d91a373dc085
    Author: Alex Seltmann <seltmann@posteo.de>
    Date:   Fri Jan 21 15:33:06 2022 +0100
    
        add correlations 100/400 clean
    
    commit e2037847eadb43f0eaaeeb48cda3cc48141e24f2
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Wed Jan 19 20:10:28 2022 +0100
    
        Fix tt_key again
    
  8. Change some parameters and execute as 2022-01-25_correlate-all-clean-ptu:
    logging.basicConfig(filename='data/exp-220120-correlate-ptu/2022-01-25_correlate-all-clean-ptu.log',
                        filemode='w', format='%(asctime)s - %(message)s',
                        force=True,
                        level=logging.DEBUG)
    weight_list = [0.2, 0.4, 0.6, 0.8]
    output_path = '/beegfs/ye53nis/saves/2022-01-25_correlate-all-clean-ptu/'
    path = Path(path)
    files = [path / f for f in os.listdir(path) if f.endswith('.ptu')]
    
    if len(files) == 0:
        raise FileNotFoundError('The path provided does not include any'
                                ' .ptu files.')
    for myfile in files:
        ptufile = cfo.PicoObject(myfile, par_obj)
        for w in weight_list:
            weight_name = f'weight{w}_{ptufile.name}'
            ptufile.getTimeSeries(timeseries_name=weight_name)
            ptufile.getPhotonCountingStats(name=weight_name)
            ptufile.predictTimeSeries(model=loaded_model,
                                      scaler='minmax',
                                      name=weight_name)
            ptufile.correctTCSPC(method='weights',
                                 weight=w,
                                 timeseries_name=weight_name)
        for key in list(ptufile.trueTimeArr.keys()):
            if "_FORWEIGHTS" in key:
                ptufile.get_autocorrelation(method='tttr2xfcs_with_weights',
                                            name=key)
    
        for m in ['multipletau', 'tttr2xfcs', 'tttr2xfcs_with_weights']:
            if m in list(ptufile.autoNorm.keys()):
                for key in list(ptufile.autoNorm[m].keys()):
                    ptufile.save_autocorrelation(name=key, method=m,
                                                 output_path=output_path)
    
    
    8e913e9c-fdcf-4a67-988c-dad76773f19c
    

2.5.5 Experiment 1b: Correlate all_dirty_ptu on Jupyter 2

  1. Make sure we are in the correct directory, also do a git log -3 to document latest git commits
    %cd /beegfs/ye53nis/drmed-git
    
    !git log -3
    
    /beegfs/ye53nis/drmed-git
    commit e2037847eadb43f0eaaeeb48cda3cc48141e24f2
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Wed Jan 19 20:10:28 2022 +0100
    
        Fix tt_key again
    
    commit 7c750e0c8ed328033d1e55b7331f5d8d8dedb4b5
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Wed Jan 19 20:07:41 2022 +0100
    
        Fix tt_key
    
    commit 6a23a927a62b1c05e3e6e438c9af4d6dc7791b48
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Wed Jan 19 20:01:15 2022 +0100
    
        Fix tttr2xfcs keyError when using standard name
    
  2. Create directory where we want to save our correlations in
    %mkdir /beegfs/ye53nis/saves/2022-01-20_correlate-all-dirty-ptu/
    
  3. Load all needed modules
    import mlflow
    import logging
    import os
    import sys
    import matplotlib.pyplot as plt
    import numpy as np
    import tensorflow as tf
    from pathlib import Path
    from pprint import pprint
    
    FLUOTRACIFY_PATH = '/beegfs/ye53nis/drmed-git/src/'
    sys.path.append(FLUOTRACIFY_PATH)
    from fluotracify.applications import corr_fit_object as cfo
    from fluotracify.training import build_model as bm
    
    logging.basicConfig(filename='data/exp-220120-correlate-ptu/2022-01-20_correlate-all-dirty-ptu.log',
                        filemode='w', format='%(asctime)s - %(message)s',
                        force=True,
                        level=logging.DEBUG)
    
    
  4. Define variables and prepare model
    class ParameterClass():
        """Stores parameters for correlation """
        def __init__(self):
            # Where the data is stored.
            self.data = []
            self.objectRef = []
            self.subObjectRef = []
            self.colors = ['blue', 'green', 'red', 'cyan', 'magenta',
                           'yellow', 'black']
            self.numOfLoaded = 0
            # very fast from Ncasc ~ 14 onwards
            self.NcascStart = 0
            self.NcascEnd = 30  # 25
            self.Nsub = 6  # 6
            self.photonLifetimeBin = 10  # used for photon decay
            self.photonCountBin = 1  # used for time series
    
    path = "../data/Pablo_structured_experiment/all_dirty_ptu"
    logged_model = 'file:///beegfs/ye53nis/drmed-git/data/mlruns/8/00f2635d9fa2463c9a066722163405be/artifacts/model'
    weight = 0
    output_path = '/beegfs/ye53nis/saves/2022-01-20_correlate-all-dirty-ptu/'
    
    par_obj = ParameterClass()
    
    loaded_model = mlflow.keras.load_model(logged_model, compile=False)
    loaded_model.compile(loss=bm.binary_ce_dice_loss(),
                         optimizer=tf.keras.optimizers.Adam(),
                         metrics = bm.unet_metrics([0.1, 0.3, 0.5, 0.7, 0.9]))
    bm.prepare_model(loaded_model)
    
    
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    
  5. Run experiment: Correlate all_dirty_ptu on Jupyter 2
    path = Path(path)
    files = [path / f for f in os.listdir(path) if f.endswith('.ptu')]
    if len(files) == 0:
        raise FileNotFoundError('The path provided does not include any'
                                ' .ptu files.')
    for myfile in files:
        ptufile = cfo.PicoObject(myfile, par_obj)
        ptufile.predictTimeSeries(model=loaded_model,
                                  scaler='minmax')
        ptufile.correctTCSPC(method='weights',
                             weight=weight)
        shift_name = f'shift_{ptufile.name}'
        ptufile.getTimeSeries(timeseries_name=shift_name)
        ptufile.getPhotonCountingStats(name=shift_name)
        ptufile.predictTimeSeries(model=loaded_model,
                                  scaler='minmax',
                                  name=shift_name)
        ptufile.correctTCSPC(method='delete_and_shift',
                             timeseries_name=shift_name)
        for key in list(ptufile.trueTimeArr.keys()):
            if "_FORWEIGHTS" in key:
                ptufile.get_autocorrelation(method='tttr2xfcs_with_weights',
                                            name=key)
            else:
                ptufile.get_autocorrelation(method='tttr2xfcs', name=key)
    
        for m in ['multipletau', 'tttr2xfcs', 'tttr2xfcs_with_weights']:
            if m in list(ptufile.autoNorm.keys()):
                for key in list(ptufile.autoNorm[m].keys()):
                    ptufile.save_autocorrelation(name=key, method=m,
                                                 output_path=output_path)
    
    
    WARNING:tensorflow:6 out of the last 9 calls to <function Model.make_predict_function.<locals>.predict_function at 0x2b217faeb9d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
    
  6. According to log, the first run ended with an Error
    [I 2022-01-23 00:26:40.512 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), keep random ports
    

    Only 229 traces were analyzed.

  7. Analyze using a different weight
    %cd /beegfs/ye53nis/drmed-git
    
    !git log -3
    
    /beegfs/ye53nis/drmed-git
    commit 9e735ecd10d0e5e5eae591f8cc40201ae6144b6a
    Author: Alex Seltmann <seltmann@posteo.de>
    Date:   Mon Jan 24 20:34:54 2022 +0100
    
        add correlations 2
    
    commit c0b4bc8494409a6b83a623977f07d91a373dc085
    Author: Alex Seltmann <seltmann@posteo.de>
    Date:   Fri Jan 21 15:33:06 2022 +0100
    
        add correlations 100/400 clean
    
    commit e2037847eadb43f0eaaeeb48cda3cc48141e24f2
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Wed Jan 19 20:10:28 2022 +0100
    
        Fix tt_key again
    
  8. Change some parameters and execute as 2022-01-25_correlate-all-dirty-ptu
    logging.basicConfig(filename='data/exp-220120-correlate-ptu/2022-01-25_correlate-all-dirty-ptu.log',
                        filemode='w', format='%(asctime)s - %(message)s',
                        force=True,
                        level=logging.DEBUG)
    weight_list = [0.2, 0.4, 0.6, 0.8]
    output_path = '/beegfs/ye53nis/saves/2022-01-25_correlate-all-dirty-ptu/'
    path = Path(path)
    files = [path / f for f in os.listdir(path) if f.endswith('.ptu')]
    
    if len(files) == 0:
        raise FileNotFoundError('The path provided does not include any'
                                ' .ptu files.')
    for myfile in files:
        ptufile = cfo.PicoObject(myfile, par_obj)
        for w in weight_list:
            weight_name = f'weight{w}_{ptufile.name}'
            ptufile.getTimeSeries(timeseries_name=weight_name)
            ptufile.getPhotonCountingStats(name=weight_name)
            ptufile.predictTimeSeries(model=loaded_model,
                                      scaler='minmax',
                                      name=weight_name)
            ptufile.correctTCSPC(method='weights',
                                 weight=w,
                                 timeseries_name=weight_name)
        for key in list(ptufile.trueTimeArr.keys()):
            if "_FORWEIGHTS" in key:
                ptufile.get_autocorrelation(method='tttr2xfcs_with_weights',
                                            name=key)
    
        for m in ['multipletau', 'tttr2xfcs', 'tttr2xfcs_with_weights']:
            if m in list(ptufile.autoNorm.keys()):
                for key in list(ptufile.autoNorm[m].keys()):
                    ptufile.save_autocorrelation(name=key, method=m,
                                                 output_path=output_path)
    
    
    fa1b6a82-0827-4730-a145-2ab9f4c967ef
    

2.5.6 Analysis 1 of no-correction vs weight=0 vs shift

  • log files can be found here:
    tail /beegfs/ye53nis/drmed-git/data/exp-220120-correlate-ptu/2022-01-20_correlate-all-clean-ptu.log
    tail /beegfs/ye53nis/drmed-git/data/exp-220120-correlate-ptu/2022-01-20_correlate-all-dirty-ptu.log
    
    (tf) [ye53nis@login01 applications]$ tail /beegfs/ye53nis/drmed-git/data/exp-220120-correlate-ptu/2022-01-20_correla
    te-all-clean-ptu.log
    2022-01-22 21:32:10,206 - Finished predictTimeSeries() with name=shift_20 nM AF488110_T1310s_1
    2022-01-22 21:32:10,249 - correctTCSPC: some samples: subChan 3486766, truetime 3486766,photonMask 3486766, channelM
    ask 3486993
    2022-01-22 21:32:10,261 - correctTCSPC: deleted 633925 photons of 3486993 photons.
    2022-01-22 21:32:10,280 - correctTCSPC: shifted non-deleted photon arrival times by photonCountBin=1
    2022-01-22 21:32:10,281 - Finished correctTCSPC() with name 20 nM AF488110_T1310s_1, timeseries_name shift_20 nM AF4
    88110_T1310s_1
    2022-01-22 21:32:10,281 - get_autocorrelation: Starting tttr2xfcs correlation.with name 20 nM AF488110_T1310s_1
    2022-01-22 21:32:10,281 - Given key 20 nM AF488110_T1310s_1 of trueTimeArr does not include a hint on which channel
    was used. Assume all channels shall be used and continue
    2022-01-22 21:32:16,525 - crossAndAuto: sum(indeces)=3486993
    2022-01-22 21:32:16,611 - crossAndAuto: finished preparation
    2022-01-22 21:32:38,406 - tttr2xfcs: finished Ncasc 0
    (tf) [ye53nis@login01 applications]$ tail /beegfs/ye53nis/drmed-git/data/exp-220120-correlate-ptu/2022-01-20_correla
    te-all-dirty-ptu.log
    2022-01-25 00:52:57,129 - Unable to restore custom metric. Please ensure that the layer implements `get_config` and
    `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    2022-01-25 00:53:05,683 - prepare_model: test shape (1, 16384, 1), e.g. [1.4064237e-01 1.3650321e-08 7.4799689e-10 1
    .7243300e-10 1.3498201e-10]
    2022-01-25 00:53:05,736 - prepare_model: test shape (1, 8192, 1). prediction failed as expected. Retry...
    2022-01-25 00:53:06,390 - prepare_model: test shape (1, 4096, 1), e.g. [1.4064237e-01 1.3650321e-08 7.4799689e-10 1.
    7243300e-10 1.3498201e-10]
    2022-01-25 00:53:06,455 - prepare_model: test shape (1, 8192, 1), e.g. [1.4064237e-01 1.3650321e-08 7.4799689e-10 1.
    7243300e-10 1.3498201e-10]
    2022-01-25 00:53:06,455 - prepare_model: UNET ready for different trace lengths
    (tf) [ye53nis@login01 applications]$
    
  • I committed the correlations to the git repository, since the file size is rather small (~5kB / correlation, makes ~15kB per file)
  • I sorted the correlations according to the 3 things I did with the .ptu data and the correlation method
    1. No correction: I just read in the .ptu data and correlated the photon arrival times using the tttr2xfcs algorithm
    2. Corrected with weights: I read in the .ptu data, constructed a timetrace from it, fed the time trace to my trained unet which predicts bright burst artifacts, then mapped the time trace prediction on the photon arrival times and gave all photons inside a bin labelled as artifactual a weight of 0 in the tttr2xfcs algorithm
    3. Corrected by photon deletion with shift: I read in the .ptu data, constructed a timetrace from it, fed the timetrace to my trained unet which predicts bright burst artifacts, then mapped the time trace prediction on the photon arrival times and deleted all photons inside a bin labelled as artifactual and the arrival times of all photons which come after a deleted bin by the bin size, then I correlated the resulting new photon arrival times using the tttr2xfcs algorithm
  • Then I fitted the traces in each of the 3 folders using Dominic Waithe's https://dwaithe.github.io/FCSfitJS/
  • now let's look at the results:
    %cd /home/lex/Programme/drmed-git
    
    /home/lex/Programme/drmed-git
    
    import numpy as np
    import matplotlib.pyplot as plt
    import pandas as pd
    import seaborn as sns
    from pathlib import Path
    from pprint import pprint
    # use seaborn style as default even if I just use matplotlib
    sns.set()
    
    folder_clean = Path('data/exp-220120-correlate-ptu/2022-01-20_correlate-all-clean-ptu')
    folder_dirty = Path('data/exp-220120-correlate-ptu/2022-01-20_correlate-all-dirty-ptu')
    clean_v1_param_path = folder_clean / '2022-01-20_clean_no-correction_param_229of400.csv'
    clean_v2_param_path = folder_clean / '2022-01-20_clean_corrected-with-weights_param_229of400.csv'
    clean_v3_param_path = folder_clean / '2022-01-20_clean_corrected-by-photon-deletion-with-shift_param_229of400.csv'
    clean_v1_plot_path = folder_clean / '2022-01-20_clean_no-correction_plot_229of400.csv'
    clean_v2_plot_path = folder_clean / '2022-01-20_clean_corrected-with-weights_plot_229of400.csv'
    clean_v3_plot_path = folder_clean / '2022-01-20_clean_corrected-by-photon-deletion-with-shift_plot_229of400.csv'
    dirty_v1_param_path = folder_dirty / '2022-01-20_dirty_no-correction_param_173of400.csv'
    dirty_v2_param_path = folder_dirty / '2022-01-20_dirty_corrected-with-weights_param_173of400.csv'
    dirty_v3_param_path = folder_dirty / '2022-01-20_dirty_corrected-by-photon-deletion-with-shift_param_173of400.csv'
    dirty_v1_plot_path = folder_dirty / '2022-01-20_dirty_no-correction_plot_173of400.csv'
    dirty_v2_plot_path = folder_dirty / '2022-01-20_dirty_corrected-with-weights_plot_173of400.csv'
    dirty_v3_plot_path = folder_dirty / '2022-01-20_dirty_corrected-by-photon-deletion-with-shift_plot_173of400.csv'
    
    
    clean_v1_param = pd.read_csv(clean_v1_param_path, sep=',')
    clean_v2_param = pd.read_csv(clean_v2_param_path, sep=',')
    clean_v3_param = pd.read_csv(clean_v3_param_path, sep=',')
    clean_v1_taudiff = clean_v1_param['txy1']
    clean_v2_taudiff = clean_v2_param['txy1']
    clean_v3_taudiff = clean_v3_param['txy1']
    dirty_v1_param = pd.read_csv(dirty_v1_param_path, sep=',')
    dirty_v2_param = pd.read_csv(dirty_v2_param_path, sep=',')
    dirty_v3_param = pd.read_csv(dirty_v3_param_path, sep=',')
    dirty_v1_taudiff = dirty_v1_param['txy1']
    dirty_v2_taudiff = dirty_v2_param['txy1']
    dirty_v3_taudiff = dirty_v3_param['txy1']
    
    display(clean_v1_param.head(2).T)
    display(clean_v2_param.head(2).T)
    display(clean_v3_param.head(2).T)
    display(dirty_v1_param.head(2).T)
    display(dirty_v2_param.head(2).T)
    display(dirty_v3_param.head(2).T)
    
      0 1
    nameofplot 2022-01-20tttr2xfcsCH2BIN120 nM AF48816T1 2022-01-20tttr2xfcsCH2BIN120 nM AF48822T2
    masterfile Not known Not known
    parentname 20 nM AF48816T182s1 20 nM AF48822T254s1
    parentuqid NaN NaN
    time of fit 24 January 2022 22:35:35 24 January 2022 22:35:35
    Diffeq Equation 1A Equation 1A
    Diffspecies 1 1
    Tripleteq Triplet Eq 2A Triplet Eq 2A
    Tripletspecies 1 1
    Dimen 3D 3D
    xmin 0.001018 0.001018
    xmax 117.440506 117.440506
    offset -0.000577 -0.000577
    stdev(offset) NaN NaN
    GN0 0.074965 0.074965
    stdev(GN0) NaN NaN
    N (FCS) 13.339549 13.339549
    cpm (kHz) 25.896822 25.862088
    A1 1 1
    stdev(A1) NaN NaN
    txy1 0.033317 0.033317
    stdev(txy1) NaN NaN
    alpha1 0.8512 0.8512
    stdev(alpha1) NaN NaN
    N (mom) 72.053415 73.936945
    bri (kHz) 4.794626 4.666217
      0 1
    nameofplot 2022-01-20tttr2xfcswithweightsCH2BIN120 2022-01-20tttr2xfcswithweightsCH2BIN120
    masterfile Not known Not known
    parentname 20 nM AF48816T182s1 20 nM AF48822T254s1
    parentuqid NaN NaN
    time of fit 24 January 2022 22:34:07 24 January 2022 22:34:07
    Diffeq Equation 1A Equation 1A
    Diffspecies 1 1
    Tripleteq Triplet Eq 2A Triplet Eq 2A
    Tripletspecies 1 1
    Dimen 3D 3D
    xmin 0.001018 0.001018
    xmax 100.66329 100.66329
    offset 0.314375 0.148697
    stdev(offset) NaN NaN
    GN0 0.161316 0.096106
    stdev(GN0) NaN NaN
    N (FCS) 6.198997 10.405181
    cpm (kHz) 55.727064 33.155463
    A1 1 1
    stdev(A1) NaN NaN
    txy1 0.066262 0.038902
    stdev(txy1) NaN NaN
    alpha1 0.5 0.737984
    stdev(alpha1) NaN NaN
    N (mom) 72.053415 73.936945
    bri (kHz) 4.794626 4.666217
      0 1
    nameofplot 2022-01-20tttr2xfcsCH2BIN1shift20 nM AF48… 2022-01-20tttr2xfcsCH2BIN1shift20 nM AF48…
    masterfile Not known Not known
    parentname shift20 nM AF48816T182s1 shift20 nM AF48822T254s1
    parentuqid NaN NaN
    time of fit 24 January 2022 22:32:21 24 January 2022 22:32:21
    Diffeq Equation 1A Equation 1A
    Diffspecies 1 1
    Tripleteq Triplet Eq 2A Triplet Eq 2A
    Tripletspecies 1 1
    Dimen 3D 3D
    xmin 0.001018 0.001018
    xmax 117.440506 117.440506
    offset 0.276905 0.139014
    stdev(offset) NaN NaN
    GN0 0.156775 0.095677
    stdev(GN0) NaN NaN
    N (FCS) 6.378581 10.451884
    cpm (kHz) 54.158119 33.007311
    A1 1 1
    stdev(A1) NaN NaN
    txy1 0.067249 0.038801
    stdev(txy1) NaN NaN
    alpha1 0.5 0.733455
    stdev(alpha1) NaN NaN
    N (mom) 72.053415 73.936945
    bri (kHz) 4.794626 4.666217
      0 1
    nameofplot 2022-01-20tttr2xfcsCH2BIN1DiO LUV 10uM in … 2022-01-20tttr2xfcsCH2BIN1DiO LUV 10uM in …
    masterfile Not known Not known
    parentname DiO LUV 10uM in 20 nM AF48823T268s1 DiO LUV 10uM in 20 nM AF48823T287s1
    parentuqid NaN NaN
    time of fit 24 January 2022 22:58:57 24 January 2022 22:59:16
    Diffeq Equation 1A Equation 1A
    Diffspecies 1 1
    Tripleteq Triplet Eq 2A Triplet Eq 2A
    Tripletspecies 1 1
    Dimen 3D 3D
    xmin 0.00089 0.00089
    xmax 2147.483642 2147.483642
    offset -0.002524 -0.014455
    stdev(offset) NaN NaN
    GN0 0.206576 0.396267
    stdev(GN0) NaN NaN
    N (FCS) 4.840844 2.523549
    cpm (kHz) 107.60087 221.540583
    A1 1 1
    stdev(A1) NaN NaN
    txy1 6.459892 4.992202
    stdev(txy1) NaN NaN
    alpha1 0.697556 0.973971
    stdev(alpha1) NaN NaN
    N (mom) 5.62743 2.845675
    bri (kHz) 92.565361 196.472342
      0 1
    nameofplot 2022-01-20tttr2xfcswithweightsCH2BIN1DiO 2022-01-20tttr2xfcswithweightsCH2BIN1DiO
    masterfile Not known Not known
    parentname DiO LUV 10uM in 20 nM AF48823T268s1 DiO LUV 10uM in 20 nM AF48823T287s1
    parentuqid NaN NaN
    time of fit 24 January 2022 22:51:01 24 January 2022 22:51:01
    Diffeq Equation 1A Equation 1A
    Diffspecies 1 1
    Tripleteq Triplet Eq 2A Triplet Eq 2A
    Tripletspecies 1 1
    Dimen 3D 3D
    xmin 0.00089 0.00089
    xmax 2415.919098 2415.919098
    offset -0.084572 -0.044295
    stdev(offset) NaN NaN
    GN0 0.338889 0.247596
    stdev(GN0) NaN NaN
    N (FCS) 2.950821 4.038831
    cpm (kHz) 176.520064 138.423335
    A1 1 1
    stdev(A1) NaN NaN
    txy1 20.767381 20.776597
    stdev(txy1) NaN NaN
    alpha1 0.5 0.5
    stdev(alpha1) NaN NaN
    N (mom) 5.62743 2.845675
    bri (kHz) 92.565361 196.472342
      0 1
    nameofplot 2022-01-20tttr2xfcsCH2BIN1shiftDiO LUV 10… 2022-01-20tttr2xfcsCH2BIN1shiftDiO LUV 10…
    masterfile Not known Not known
    parentname shiftDiO LUV 10uM in 20 nM AF48823T268s1 shiftDiO LUV 10uM in 20 nM AF48823T287s1
    parentuqid NaN NaN
    time of fit 24 January 2022 23:05:52 24 January 2022 23:06:00
    Diffeq Equation 1A Equation 1A
    Diffspecies 1 1
    Tripleteq Triplet Eq 2A Triplet Eq 2A
    Tripletspecies 1 1
    Dimen 3D 3D
    xmin 0.00089 0.00089
    xmax 2415.919098 2415.919098
    offset -0.023485 0.011508
    stdev(offset) NaN NaN
    GN0 0.304118 0.211213
    stdev(GN0) NaN NaN
    N (FCS) 3.288199 4.734564
    cpm (kHz) 158.408619 118.082341
    A1 1 1
    stdev(A1) NaN NaN
    txy1 2.884429 1.956253
    stdev(txy1) NaN NaN
    alpha1 0.5 0.600835
    stdev(alpha1) NaN NaN
    N (mom) 5.62743 2.845675
    bri (kHz) 92.565361 196.472342
    clean_v1_plot = pd.read_csv(clean_v1_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 459', axis=1)
    clean_v2_plot = pd.read_csv(clean_v2_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 459', axis=1)
    clean_v3_plot = pd.read_csv(clean_v3_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 459', axis=1)
    clean_v1_tau = clean_v1_plot['Time (ms)']
    clean_v2_tau = clean_v2_plot['Time (ms)']
    clean_v3_tau = clean_v3_plot['Time (ms)']
    clean_v1_corr = clean_v1_plot.iloc[:, 1::2]
    clean_v2_corr = clean_v2_plot.iloc[:, 1::2]
    clean_v3_corr = clean_v3_plot.iloc[:, 1::2]
    clean_v1_fit = clean_v1_plot.iloc[:, 2::2]
    clean_v2_fit = clean_v2_plot.iloc[:, 2::2]
    clean_v3_fit = clean_v3_plot.iloc[:, 2::2]
    dirty_v1_plot = pd.read_csv(dirty_v1_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 347', axis=1)
    dirty_v2_plot = pd.read_csv(dirty_v2_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 347', axis=1)
    dirty_v3_plot = pd.read_csv(dirty_v3_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 347', axis=1)
    dirty_v1_tau = dirty_v1_plot['Time (ms)']
    dirty_v2_tau = dirty_v2_plot['Time (ms)']
    dirty_v3_tau = dirty_v3_plot['Time (ms)']
    dirty_v1_corr = dirty_v1_plot.iloc[:, 1::2]
    dirty_v2_corr = dirty_v2_plot.iloc[:, 1::2]
    dirty_v3_corr = dirty_v3_plot.iloc[:, 1::2]
    dirty_v1_fit = dirty_v1_plot.iloc[:, 2::2]
    dirty_v2_fit = dirty_v2_plot.iloc[:, 2::2]
    dirty_v3_fit = dirty_v3_plot.iloc[:, 2::2]
    
    taudiff = pd.DataFrame(data=[clean_v1_taudiff, clean_v2_taudiff, clean_v3_taudiff,
                                 dirty_v1_taudiff, dirty_v2_taudiff, dirty_v3_taudiff],
                           index=['clean-no c.', 'clean-weights', 'clean-shift',
                                  'dirty-no c.', 'dirty-weights', 'dirty-shift']).T
    fig = plt.figure(figsize=(16,9))
    fig = sns.boxplot(data=taudiff, showfliers=False)
    ylims = fig.get_ylim()
    fig.set(ylim=ylims)
    fig = sns.stripplot(data=taudiff, jitter=True, color='.3')
    plt.show()
    

    analysis1-nocorr+w0+shift.png

  • So from the first run we see:
    1. we confirm the distortion of transit times in the dirty data
    2. we see that the correction methods weight=0 and delete_and_shift as they are implemented now don't work

2.5.7 Analysis 2 of weight=0.2…0.8

  • log files can be found here:
    tail /beegfs/ye53nis/drmed-git/data/exp-220120-correlate-ptu/2022-01-25_correlate-all-clean-ptu.log
    tail /beegfs/ye53nis/drmed-git/data/exp-220120-correlate-ptu/2022-01-25_correlate-all-dirty-ptu.log
    
    (tf) [ye53nis@login01 applications]$ tail /beegfs/ye53nis/drmed-git/data/exp-220120-correlate-ptu/2022-01-25_correla te-all-clean-ptu.log
    2022-01-26 15:58:00,327 - Finished predictTimeSeries() with name=weight0.6_20 nM AF48866_T782s_1
    2022-01-26 15:58:00,367 - correctTCSPC: some samples: subChan 3437400, truetime 3437400,photonMask 3437400, channelM ask 3437580
    2022-01-26 15:58:00,409 - Finished correctTCSPC() with name 20 nM AF48866_T782s_1, timeseries_name weight0.6_20 nM A F48866_T782s_1
    2022-01-26 15:58:00,513 - Finished time2bin. last_time=9995.0, num_bins=9995.0
    2022-01-26 15:58:00,514 - Finished getTimeSeries() with truetime_name 20 nM AF48866_T782s_1, timeseries_name weight0 .8_20 nM AF48866_T782s_1
    2022-01-26 15:58:00,516 - Finished getPhotonCountingStats() with name: weight0.8_20 nM AF48866_T782s_1
    2022-01-26 15:58:00,568 - Finished predictTimeSeries() with name=weight0.8_20 nM AF48866_T782s_1
    2022-01-26 15:58:00,607 - correctTCSPC: some samples: subChan 3437400, truetime 3437400,photonMask 3437400, channelM ask 3437580
    2022-01-26 15:58:00,668 - Finished correctTCSPC() with name 20 nM AF48866_T782s_1, timeseries_name weight0.8_20 nM A F48866_T782s_1
    2022-01-26 15:58:00,668 - get_autocorrelation: Starting tttr2xfcs correlation.with name CH2_weight0.2_20 nM AF48866_ T782s_1_FORWEIGHTS
    (tf) [ye53nis@login01 applications]$ tail /beegfs/ye53nis/drmed-git/data/exp-220120-correlate-ptu/2022-01-25_correla te-all-dirty-ptu.log
    2022-01-26 15:43:14,278 - Finished predictTimeSeries() with name=weight0.6_DiO LUV 10uM in 20 nM AF48876_T910s_1
    2022-01-26 15:43:14,406 - correctTCSPC: some samples: subChan 4672383, truetime 4672383,photonMask 4672383, channelM ask 4672675
    2022-01-26 15:43:14,520 - Finished correctTCSPC() with name DiO LUV 10uM in 20 nM AF48876_T910s_1, timeseries_name w eight0.6_DiO LUV 10uM in 20 nM AF48876_T910s_1
    2022-01-26 15:43:14,696 - Finished time2bin. last_time=9996.0, num_bins=9996.0
    2022-01-26 15:43:14,705 - Finished getTimeSeries() with truetime_name DiO LUV 10uM in 20 nM AF48876_T910s_1, timeser ies_name weight0.8_DiO LUV 10uM in 20 nM AF48876_T910s_1
    2022-01-26 15:43:14,707 - Finished getPhotonCountingStats() with name: weight0.8_DiO LUV 10uM in 20 nM AF48876_T910s _1
    2022-01-26 15:43:14,762 - Finished predictTimeSeries() with name=weight0.8_DiO LUV 10uM in 20 nM AF48876_T910s_1
    2022-01-26 15:43:14,889 - correctTCSPC: some samples: subChan 4672383, truetime 4672383,photonMask 4672383, channelM ask 4672675
    2022-01-26 15:43:14,960 - Finished correctTCSPC() with name DiO LUV 10uM in 20 nM AF48876_T910s_1, timeseries_name w eight0.8_DiO LUV 10uM in 20 nM AF48876_T910s_1
    2022-01-26 15:43:14,963 - get_autocorrelation: Starting tttr2xfcs correlation.with name CH2_weight0.2_DiO LUV 10uM i n 20 nM AF48876_T910s_1_FORWEIGHTS
    (tf) [ye53nis@login01 applications]$
    
  • So the first analysis of "no correction" vs "weight=0" vs "delete and shift" did show that the two correction methods did not work properly. Another approach is photon weighting, so let's look at the following correlations:
    1. no correction: I just read in the .ptu data and correlated the photon arrival times using the tttr2xfcs algorithm → taken from first experiment
    2. weight=0: I read in the .ptu data, constructed a timetrace from it, fed the time trace to my trained unet which predicts bright burst artifacts, then mapped the time trace prediction on the photon arrival times and gave all photons inside a bin labelled as artifactual a weight of 0.2 in the tttr2xfcs algorithm
    3. weight=0.2: changed weight to 0.2, else see above
    4. weight=0.4: changed weight to 0.4, else see above
    5. weight=0.6: changed weight to 0.6, else see above
    6. weight=0.8: changed weight to 0.8, else see above
  • Then I fitted the traces in each of the 3 folders using Dominic Waithe's https://dwaithe.github.io/FCSfitJS/
  • now let's look at the results:
    %cd /home/lex/Programme/drmed-git
    
    /home/lex/Programme/drmed-git
    
    import numpy as np
    import matplotlib.pyplot as plt
    import pandas as pd
    import seaborn as sns
    from pathlib import Path
    from pprint import pprint
    # use seaborn style as default even if I just use matplotlib
    sns.set()
    
    folder_clean = Path('data/exp-220120-correlate-ptu/2022-01-20_correlate-all-clean-ptu')
    folder_clean_weights = Path('data/exp-220120-correlate-ptu/2022-01-25_correlate-all-clean-ptu')
    folder_dirty = Path('data/exp-220120-correlate-ptu/2022-01-20_correlate-all-dirty-ptu')
    folder_dirty_weights = Path('data/exp-220120-correlate-ptu/2022-01-25_correlate-all-dirty-ptu')
    clean_v1_param_path = folder_clean / '2022-01-20_clean_no-correction_param_229of400.csv'
    clean_00_param_path = folder_clean / '2022-01-20_clean_corrected-with-weights_param_229of400.csv'
    clean_02_param_path = folder_clean_weights / '2022-01-25_clean_corrected-with-weights-0.2_param_34of400.csv'
    clean_04_param_path = folder_clean_weights / '2022-01-25_clean_corrected-with-weights-0.4_param_34of400.csv'
    clean_06_param_path = folder_clean_weights / '2022-01-25_clean_corrected-with-weights-0.6_param_34of400.csv'
    clean_08_param_path = folder_clean_weights / '2022-01-25_clean_corrected-with-weights-0.8_param_34of400.csv'
    
    clean_v1_plot_path = folder_clean / '2022-01-20_clean_no-correction_plot_229of400.csv'
    clean_00_plot_path = folder_clean / '2022-01-20_clean_corrected-with-weights_plot_229of400.csv'
    clean_02_plot_path = folder_clean_weights / '2022-01-25_clean_corrected-with-weights-0.2_plot_34of400.csv'
    clean_04_plot_path = folder_clean_weights / '2022-01-25_clean_corrected-with-weights-0.4_plot_34of400.csv'
    clean_06_plot_path = folder_clean_weights / '2022-01-25_clean_corrected-with-weights-0.6_plot_34of400.csv'
    clean_08_plot_path = folder_clean_weights / '2022-01-25_clean_corrected-with-weights-0.8_plot_34of400.csv'
    
    dirty_v1_param_path = folder_dirty / '2022-01-20_dirty_no-correction_param_173of400.csv'
    dirty_00_param_path = folder_dirty / '2022-01-20_dirty_corrected-with-weights_param_173of400.csv'
    dirty_02_param_path = folder_dirty_weights / '2022-01-25_dirty_corrected-with-weights-0.2_param_23of400.csv'
    dirty_04_param_path = folder_dirty_weights / '2022-01-25_dirty_corrected-with-weights-0.4_param_23of400.csv'
    dirty_06_param_path = folder_dirty_weights / '2022-01-25_dirty_corrected-with-weights-0.6_param_23of400.csv'
    dirty_08_param_path = folder_dirty_weights / '2022-01-25_dirty_corrected-with-weights-0.8_param_23of400.csv'
    
    dirty_v1_plot_path = folder_dirty / '2022-01-20_dirty_no-correction_plot_173of400.csv'
    dirty_00_plot_path = folder_dirty / '2022-01-20_dirty_corrected-with-weights_plot_173of400.csv'
    dirty_02_plot_path = folder_dirty_weights / '2022-01-25_dirty_corrected-with-weights-0.2_plot_23of400.csv'
    dirty_04_plot_path = folder_dirty_weights / '2022-01-25_dirty_corrected-with-weights-0.4_plot_23of400.csv'
    dirty_06_plot_path = folder_dirty_weights / '2022-01-25_dirty_corrected-with-weights-0.6_plot_23of400.csv'
    dirty_08_plot_path = folder_dirty_weights / '2022-01-25_dirty_corrected-with-weights-0.8_plot_23of400.csv'
    
    
    clean_v1_param = pd.read_csv(clean_v1_param_path, sep=',')
    clean_00_param = pd.read_csv(clean_02_param_path, sep=',')
    clean_02_param = pd.read_csv(clean_02_param_path, sep=',')
    clean_04_param = pd.read_csv(clean_04_param_path, sep=',')
    clean_06_param = pd.read_csv(clean_06_param_path, sep=',')
    clean_08_param = pd.read_csv(clean_08_param_path, sep=',')
    
    clean_v1_taudiff = clean_v1_param['txy1']
    clean_00_taudiff = clean_02_param['txy1']
    clean_02_taudiff = clean_02_param['txy1']
    clean_04_taudiff = clean_04_param['txy1']
    clean_06_taudiff = clean_06_param['txy1']
    clean_08_taudiff = clean_08_param['txy1']
    
    dirty_v1_param = pd.read_csv(dirty_v1_param_path, sep=',')
    dirty_00_param = pd.read_csv(dirty_02_param_path, sep=',')
    dirty_02_param = pd.read_csv(dirty_02_param_path, sep=',')
    dirty_04_param = pd.read_csv(dirty_04_param_path, sep=',')
    dirty_06_param = pd.read_csv(dirty_06_param_path, sep=',')
    dirty_08_param = pd.read_csv(dirty_08_param_path, sep=',')
    
    dirty_v1_taudiff = dirty_v1_param['txy1']
    dirty_00_taudiff = dirty_02_param['txy1']
    dirty_02_taudiff = dirty_02_param['txy1']
    dirty_04_taudiff = dirty_04_param['txy1']
    dirty_06_taudiff = dirty_06_param['txy1']
    dirty_08_taudiff = dirty_08_param['txy1']
    
    display(pd.concat([clean_v1_param.head(1).T, clean_00_param.head(1).T, clean_02_param.head(1).T,
               clean_04_param.head(1).T, clean_06_param.head(1).T, clean_08_param.head(1).T], axis=1))
    display(pd.concat([dirty_v1_param.head(1).T, dirty_00_param.head(1).T, dirty_02_param.head(1).T,
               dirty_04_param.head(1).T, dirty_06_param.head(1).T, dirty_08_param.head(1).T], axis=1))
    
    
      0 0 0 0 0 0
    nameofplot 2022-01-20tttr2xfcsCH2BIN120 nM AF48816T1 2022-01-25tttr2xfcswithweightsCH2BIN1wei 2022-01-25tttr2xfcswithweightsCH2BIN1wei 2022-01-25tttr2xfcswithweightsCH2BIN1wei 2022-01-25tttr2xfcswithweightsCH2BIN1wei 2022-01-25tttr2xfcswithweightsCH2BIN1wei
    masterfile Not known Not known Not known Not known Not known Not known
    parentname 20 nM AF48816T182s1 weight0.220 nM AF48816T182s1 weight0.220 nM AF48816T182s1 weight0.420 nM AF48816T182s1 weight0.620 nM AF48816T182s1 weight0.820 nM AF48816T182s1
    parentuqid NaN NaN NaN NaN NaN NaN
    time of fit 24 January 2022 22:35:35 25 January 2022 14:14:37 25 January 2022 14:14:37 25 January 2022 14:17:36 25 January 2022 14:18:57 25 January 2022 14:20:30
    Diffeq Equation 1A Equation 1A Equation 1A Equation 1A Equation 1A Equation 1A
    Diffspecies 1 1 1 1 1 1
    Tripleteq Triplet Eq 2A Triplet Eq 2A Triplet Eq 2A Triplet Eq 2A Triplet Eq 2A Triplet Eq 2A
    Tripletspecies 1 1 1 1 1 1
    Dimen 3D 3D 3D 3D 3D 3D
    xmin 0.001018 0.001018 0.001018 0.001018 0.001146 0.001146
    xmax 117.440506 1073.741818 1073.741818 469.762042 369.098746 603.97977
    offset -0.000577 0.171278 0.171278 0.087402 0.035123 0.007769
    stdev(offset) NaN NaN NaN NaN NaN NaN
    GN0 0.074965 0.132934 0.132934 0.095521 0.078695 0.073745
    stdev(GN0) NaN NaN NaN NaN NaN NaN
    N (FCS) 13.339549 7.522504 7.522504 10.468952 12.70734 13.560257
    cpm (kHz) 25.896822 45.922465 45.922465 32.997757 27.185228 25.475323
    A1 1 1 1 1 1 1
    stdev(A1) NaN NaN NaN NaN NaN NaN
    txy1 0.033317 0.042671 0.042671 0.03675 0.034622 0.033477
    stdev(txy1) NaN NaN NaN NaN NaN NaN
    alpha1 0.8512 0.5 0.5 0.691478 0.843979 0.887953
    stdev(alpha1) NaN NaN NaN NaN NaN NaN
    N (mom) 72.053415 72.053415 72.053415 72.053415 72.053415 72.053415
    bri (kHz) 4.794626 4.794626 4.794626 4.794626 4.794626 4.794626
      0 0 0 0 0 0
    nameofplot 2022-01-20tttr2xfcsCH2BIN1DiO LUV 10uM in … 2022-01-25tttr2xfcswithweightsCH2BIN1wei 2022-01-25tttr2xfcswithweightsCH2BIN1wei 2022-01-25tttr2xfcswithweightsCH2BIN1wei 2022-01-25tttr2xfcswithweightsCH2BIN1wei 2022-01-25tttr2xfcswithweightsCH2BIN1wei
    masterfile Not known Not known Not known Not known Not known Not known
    parentname DiO LUV 10uM in 20 nM AF48823T268s1 weight0.2DiO LUV 10uM in 20 nM AF48823T268s1 weight0.2DiO LUV 10uM in 20 nM AF48823T268s1 weight0.4DiO LUV 10uM in 20 nM AF48823T268s1 weight0.6DiO LUV 10uM in 20 nM AF48823T268s1 weight0.8DiO LUV 10uM in 20 nM AF48823T268s1
    parentuqid NaN NaN NaN NaN NaN NaN
    time of fit 24 January 2022 22:58:57 25 January 2022 13:58:13 25 January 2022 13:58:13 25 January 2022 14:03:08 25 January 2022 14:04:21 25 January 2022 14:07:40
    Diffeq Equation 1A Equation 1A Equation 1A Equation 1A Equation 1A Equation 1A
    Diffspecies 1 1 1 1 1 1
    Tripleteq Triplet Eq 2A Triplet Eq 2A Triplet Eq 2A Triplet Eq 2A Triplet Eq 2A Triplet Eq 2A
    Tripletspecies 1 1 1 1 1 1
    Dimen 3D 3D 3D 3D 3D 3D
    xmin 0.00089 0.00089 0.00089 0.001018 0.001018 0.001018
    xmax 2147.483642 1073.741818 1073.741818 1610.61273 1610.61273 1879.048186
    offset -0.002524 -0.004259 -0.004259 0.001018 0.000758 -0.002585
    stdev(offset) NaN NaN NaN NaN NaN NaN
    GN0 0.206576 0.149601 0.149601 0.092027 0.095303 0.138446
    stdev(GN0) NaN NaN NaN NaN NaN NaN
    N (FCS) 4.840844 6.684437 6.684437 10.866429 10.492811 7.223025
    cpm (kHz) 107.60087 77.924146 77.924146 47.934705 49.641517 72.113704
    A1 1 1 1 1 1 1
    stdev(A1) NaN NaN NaN NaN NaN NaN
    txy1 6.459892 0.811509 0.811509 0.309739 0.802572 3.675292
    stdev(txy1) NaN NaN NaN NaN NaN NaN
    alpha1 0.697556 0.5 0.5 0.5277 0.5 0.567203
    stdev(alpha1) NaN NaN NaN NaN NaN NaN
    N (mom) 5.62743 5.62743 5.62743 5.62743 5.62743 5.62743
    bri (kHz) 92.565361 92.565361 92.565361 92.565361 92.565361 92.565361
    clean_v1_plot = pd.read_csv(clean_v1_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 459', axis=1)
    clean_00_plot = pd.read_csv(clean_00_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 459', axis=1)
    clean_02_plot = pd.read_csv(clean_02_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 69', axis=1)
    clean_04_plot = pd.read_csv(clean_04_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 69', axis=1)
    clean_06_plot = pd.read_csv(clean_06_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 69', axis=1)
    clean_08_plot = pd.read_csv(clean_08_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 69', axis=1)
    
    clean_v1_tau = clean_v1_plot['Time (ms)']
    clean_00_tau = clean_00_plot['Time (ms)']
    clean_02_tau = clean_02_plot['Time (ms)']
    clean_04_tau = clean_04_plot['Time (ms)']
    clean_06_tau = clean_06_plot['Time (ms)']
    clean_08_tau = clean_08_plot['Time (ms)']
    clean_v1_corr = clean_v1_plot.iloc[:, 1::2]
    clean_00_corr = clean_00_plot.iloc[:, 1::2]
    clean_02_corr = clean_02_plot.iloc[:, 1::2]
    clean_04_corr = clean_04_plot.iloc[:, 1::2]
    clean_06_corr = clean_06_plot.iloc[:, 1::2]
    clean_08_corr = clean_08_plot.iloc[:, 1::2]
    clean_v1_fit = clean_v1_plot.iloc[:, 2::2]
    clean_00_fit = clean_00_plot.iloc[:, 2::2]
    clean_02_fit = clean_02_plot.iloc[:, 2::2]
    clean_04_fit = clean_04_plot.iloc[:, 2::2]
    clean_06_fit = clean_06_plot.iloc[:, 2::2]
    clean_08_fit = clean_08_plot.iloc[:, 2::2]
    dirty_v1_plot = pd.read_csv(dirty_v1_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 347', axis=1)
    dirty_00_plot = pd.read_csv(dirty_00_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 347', axis=1)
    dirty_02_plot = pd.read_csv(dirty_02_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 47', axis=1)
    dirty_04_plot = pd.read_csv(dirty_04_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 47', axis=1)
    dirty_06_plot = pd.read_csv(dirty_06_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 47', axis=1)
    dirty_08_plot = pd.read_csv(dirty_08_plot_path, sep=',', na_values=' ').drop(
        'Unnamed: 47', axis=1)
    dirty_v1_tau = dirty_v1_plot['Time (ms)']
    dirty_00_tau = dirty_00_plot['Time (ms)']
    dirty_02_tau = dirty_02_plot['Time (ms)']
    dirty_04_tau = dirty_04_plot['Time (ms)']
    dirty_06_tau = dirty_06_plot['Time (ms)']
    dirty_08_tau = dirty_08_plot['Time (ms)']
    dirty_v1_corr = dirty_v1_plot.iloc[:, 1::2]
    dirty_00_corr = dirty_00_plot.iloc[:, 1::2]
    dirty_02_corr = dirty_02_plot.iloc[:, 1::2]
    dirty_04_corr = dirty_04_plot.iloc[:, 1::2]
    dirty_06_corr = dirty_06_plot.iloc[:, 1::2]
    dirty_08_corr = dirty_08_plot.iloc[:, 1::2]
    dirty_v1_fit = dirty_v1_plot.iloc[:, 2::2]
    dirty_00_fit = dirty_00_plot.iloc[:, 2::2]
    dirty_02_fit = dirty_02_plot.iloc[:, 2::2]
    dirty_04_fit = dirty_04_plot.iloc[:, 2::2]
    dirty_06_fit = dirty_06_plot.iloc[:, 2::2]
    dirty_08_fit = dirty_08_plot.iloc[:, 2::2]
    
    
    taudiff = pd.DataFrame(data=[clean_v1_taudiff, clean_00_taudiff, clean_02_taudiff,
                                 clean_04_taudiff, clean_06_taudiff, clean_08_taudiff,
                                 dirty_v1_taudiff, dirty_00_taudiff, dirty_02_taudiff,
                                 dirty_04_taudiff, dirty_06_taudiff, dirty_08_taudiff],
                           index=['clean|no c.', 'clean|weight=0', 'clean|weight=0.2',
                                  'clean|weight=0.4', 'clean|weight=0.6', 'clean|weight=0.8',
                                  'dirty|no c.', 'dirty|weight=0', 'dirty|weight=0.2',
                                  'dirty|weight=0.4', 'dirty|weight=0.6', 'dirty|weight=0.8']).T
    fig, ax = plt.subplots(figsize=(12,5))
    #ax.set_yscale('log')
    fig = sns.boxplot(data=taudiff, showfliers=False)
    ylims = fig.get_ylim()
    fig.set(ylim=ylims)
    fig = sns.stripplot(data=taudiff, jitter=True, color='.3')
    plt.xticks(rotation=45)
    plt.tight_layout()
    plt.show()
    
  • Now on the first glance with a linear y-scale the correction by weights seemed to work with w=0.4, but we have to look at the log plot and see that this correction is not yet sufficient:
    • analysis2-allclean-alldirty.png
    • analysis2-allclean-alldirty-log.png
    taudiff = pd.DataFrame(data=[clean_v1_taudiff, clean_00_taudiff, clean_02_taudiff,
                                 clean_04_taudiff, clean_06_taudiff, clean_08_taudiff,
                                 dirty_04_taudiff, dirty_v1_taudiff],
                           index=['clean|no c.', 'clean|weight=0', 'clean|weight=0.2',
                                  'clean|weight=0.4', 'clean|weight=0.6', 'clean|weight=0.8',
                                  'dirty|weight=0.4', 'dirty|no c.']).T
    fig, ax = plt.subplots(figsize=(9,6))
    ax.set_yscale('log')
    fig = sns.boxplot(data=taudiff, showfliers=False)
    ylims = fig.get_ylim()
    fig.set(ylim=ylims)
    fig = sns.stripplot(data=taudiff, jitter=True, color='.3')
    plt.xticks(rotation=45)
    plt.tight_layout()
    plt.show()
    
  • While we are at it let's look at some subsamples of the data:
  • Here are only the clean plots:
    • analysis2-clean-log.png
  • Here are the clean controls and the dirty data with the best correction so far (w=0.4) and the dirty data without correction
    • analysis2-clean+dirty04+dirtync-log.png
  • What are learnings so far?
    • I need to optimize the correlation speed, e.g. by implementing the Cython version of the tttr2xfcs correlation
    • we have to investigate why the correction by weights doesn't work
  • Dominic actually had a look at the data and also looked at 2 component fitting:
      Fast fraction   Slow fraction  
    Weights txy1 (ms) A1 txy2 (ms) A2
    0.0 0.2611 0.6973 25.4920 0.8459
    0.2000 0.2222 0.8455 19.3756 0.5863
    0.4000 0.0446 0.6774 3.5694 0.8409
    0.6000 0.0492 0.5210 5.7868 0.9703
    0.8000 0.0504 0.3787 9.1164 0.9561
             
    DIRTY 0.0717 0.2569 12.0547 0.9740
    CLEAN 0.0368 0.9924 283.0306 0.0368
  • he used a 2 component model, varied A1 and A2, but kept alpha1-2 fixed at 1.0.
  • the data suggest that reducing the weight, it gets better till 0.4, whereafter some funny things happen with the data.
    • we should analyze with random weights to see if there is an artefact with setting the weight specifically to 0 or 0.2.
  • also it is surprising to see that 2 component fitting works so well.
  • it is interesting to see that A1 gets larger for txy1 for smaller weights, showing this component is becoming enriched which is good (until 0.0 which is strange) and conversely for A2
  • FROM HINDSIGHT: Dominic used a 2D fit here - but we have a 3D situation here and this table is so far not reproducible in 3D with equation 1B

2.5.8 Experiment 2a: Update packages and Correlate all_clean_ptu on Jupyter 1

  1. Make sure we are in the correct directory, also do a git log -3 to document latest git commits
    %cd /beegfs/ye53nis/drmed-git
    
    !git log -3
    
    /beegfs/ye53nis/drmed-git
    commit dffd95906e760976dfdfd635b50cf056c8813ca3
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Wed Feb 9 23:24:34 2022 +0100
    
        Fix memory leak by deleting variables 2
    
    commit 298240ae97e683133071500ff83e0dbd2434883e
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Wed Feb 9 23:20:32 2022 +0100
    
        Fix memory leak by deleting variables
    commit b2987fdc6615e71118ce28f47bc5174109c72ff1
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Tue Feb 8 17:24:12 2022 +0100
    
        Fix get_autocorrelation tttr2xfcs_with_weights
    
  2. Create directory where we want to save our correlations in
    %mkdir /beegfs/ye53nis/saves/2022-02-08_correlate-all-clean-ptu/
    
  3. Install dividAndConquer as a Cython implementation to make tttr2xfcs a lot faster. First, update all packages, because of a deprecated NumPy API
    cd /beegfs/ye53nis/drmed-git/
    # executed interactively:
    # conda env remove -n tf
    # conda activate tf
    # pip install numpy mlflow lmfit scikit-learn tensorflow matplotlib pandas
    #     seaborn f csfiles multipletau cython
    # pip install cython --no-binary cython
    
    (base) [ye53nis@login01 drmed-git]$ conda env remove -n tf
    
    Remove all packages in environment /home/ye53nis/.conda/envs/tf:
    
    (base) [ye53nis@login01 drmed-git]$ conda create -n tf jupyterlab pip
    Collecting package metadata (current_repodata.json): done
    Solving environment: done
    
    
    ==> WARNING: A newer version of conda exists. <==
      current version: 4.10.0
      latest version: 4.11.0
    
    Please update conda by running
    
        $ conda update -n base -c defaults conda
    
    
    
    ## Package Plan ##
    
      environment location: /home/ye53nis/.conda/envs/tf
    
      added / updated specs:
        - jupyterlab
        - pip
    
    
    The following packages will be downloaded:
    
        package                    |            build
        ---------------------------|-----------------
        attrs-21.4.0               |     pyhd3eb1b0_0          51 KB
        bleach-4.1.0               |     pyhd3eb1b0_0         123 KB
        ca-certificates-2021.10.26 |       h06a4308_2         115 KB
        certifi-2021.10.8          |   py39h06a4308_2         151 KB
        cffi-1.15.0                |   py39hd667e15_1         225 KB
        charset-normalizer-2.0.4   |     pyhd3eb1b0_0          35 KB
        cryptography-36.0.0        |   py39h9ce1e76_0         1.3 MB
        debugpy-1.5.1              |   py39h295c915_0         1.7 MB
        decorator-5.1.1            |     pyhd3eb1b0_0          12 KB
        idna-3.3                   |     pyhd3eb1b0_0          49 KB
        importlib-metadata-4.8.2   |   py39h06a4308_0          39 KB
        importlib_metadata-4.8.2   |       hd3eb1b0_0          12 KB
        ipykernel-6.4.1            |   py39h06a4308_1         194 KB
        ipython-7.31.1             |   py39h06a4308_0        1006 KB
        jedi-0.18.1                |   py39h06a4308_1         982 KB
        jinja2-3.0.2               |     pyhd3eb1b0_0         110 KB
        jsonschema-3.2.0           |     pyhd3eb1b0_2          47 KB
        jupyter_client-7.1.2       |     pyhd3eb1b0_0          93 KB
        jupyter_core-4.9.1         |   py39h06a4308_0          75 KB
        jupyterlab-3.2.1           |     pyhd3eb1b0_1         3.6 MB
        jupyterlab_server-2.10.2   |     pyhd3eb1b0_1          48 KB
        matplotlib-inline-0.1.2    |     pyhd3eb1b0_2          12 KB
        nbconvert-6.3.0            |   py39h06a4308_0         488 KB
        ncurses-6.3                |       h7f8727e_2         782 KB
        notebook-6.4.6             |   py39h06a4308_0         4.2 MB
        openssl-1.1.1m             |       h7f8727e_0         2.5 MB
        packaging-21.3             |     pyhd3eb1b0_0          36 KB
        pandocfilters-1.5.0        |     pyhd3eb1b0_0          11 KB
        parso-0.8.3                |     pyhd3eb1b0_0          70 KB
        pip-21.2.4                 |   py39h06a4308_0         1.8 MB
        prometheus_client-0.13.1   |     pyhd3eb1b0_0          47 KB
        prompt-toolkit-3.0.20      |     pyhd3eb1b0_0         259 KB
        pycparser-2.21             |     pyhd3eb1b0_0          94 KB
        pygments-2.11.2            |     pyhd3eb1b0_0         759 KB
        pyopenssl-22.0.0           |     pyhd3eb1b0_0          50 KB
        pyparsing-3.0.4            |     pyhd3eb1b0_0          81 KB
        pyrsistent-0.18.0          |   py39heee7806_0          94 KB
        python-3.9.7               |       h12debd9_1        18.6 MB
        pytz-2021.3                |     pyhd3eb1b0_0         171 KB
        pyzmq-22.3.0               |   py39h295c915_2         470 KB
        readline-8.1.2             |       h7f8727e_1         354 KB
        requests-2.27.1            |     pyhd3eb1b0_0          54 KB
        send2trash-1.8.0           |     pyhd3eb1b0_1          19 KB
        setuptools-58.0.4          |   py39h06a4308_0         790 KB
        sqlite-3.37.2              |       hc218d9a_0        1008 KB
        tk-8.6.11                  |       h1ccaba5_0         3.0 MB
        traitlets-5.1.1            |     pyhd3eb1b0_0          84 KB
        tzdata-2021e               |       hda174b7_0         112 KB
        urllib3-1.26.8             |     pyhd3eb1b0_0         106 KB
        wcwidth-0.2.5              |     pyhd3eb1b0_0          26 KB
        wheel-0.37.1               |     pyhd3eb1b0_0          33 KB
        zipp-3.7.0                 |     pyhd3eb1b0_0          12 KB
        zlib-1.2.11                |       h7f8727e_4         108 KB
        ------------------------------------------------------------
                                               Total:        46.0 MB
    
    The following NEW packages will be INSTALLED:
    
      _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
      _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-4.5-1_gnu
      anyio              pkgs/main/linux-64::anyio-2.2.0-py39h06a4308_1
      argon2-cffi        pkgs/main/linux-64::argon2-cffi-20.1.0-py39h27cfd23_1
      async_generator    pkgs/main/noarch::async_generator-1.10-pyhd3eb1b0_0
      attrs              pkgs/main/noarch::attrs-21.4.0-pyhd3eb1b0_0
      babel              pkgs/main/noarch::babel-2.9.1-pyhd3eb1b0_0
      backcall           pkgs/main/noarch::backcall-0.2.0-pyhd3eb1b0_0
      bleach             pkgs/main/noarch::bleach-4.1.0-pyhd3eb1b0_0
      brotlipy           pkgs/main/linux-64::brotlipy-0.7.0-py39h27cfd23_1003
      ca-certificates    pkgs/main/linux-64::ca-certificates-2021.10.26-h06a4308_2
      certifi            pkgs/main/linux-64::certifi-2021.10.8-py39h06a4308_2
      cffi               pkgs/main/linux-64::cffi-1.15.0-py39hd667e15_1
      charset-normalizer pkgs/main/noarch::charset-normalizer-2.0.4-pyhd3eb1b0_0
      cryptography       pkgs/main/linux-64::cryptography-36.0.0-py39h9ce1e76_0
      debugpy            pkgs/main/linux-64::debugpy-1.5.1-py39h295c915_0
      decorator          pkgs/main/noarch::decorator-5.1.1-pyhd3eb1b0_0
      defusedxml         pkgs/main/noarch::defusedxml-0.7.1-pyhd3eb1b0_0
      entrypoints        pkgs/main/linux-64::entrypoints-0.3-py39h06a4308_0
      idna               pkgs/main/noarch::idna-3.3-pyhd3eb1b0_0
      importlib-metadata pkgs/main/linux-64::importlib-metadata-4.8.2-py39h06a4308_0
      importlib_metadata pkgs/main/noarch::importlib_metadata-4.8.2-hd3eb1b0_0
      ipykernel          pkgs/main/linux-64::ipykernel-6.4.1-py39h06a4308_1
      ipython            pkgs/main/linux-64::ipython-7.31.1-py39h06a4308_0
      ipython_genutils   pkgs/main/noarch::ipython_genutils-0.2.0-pyhd3eb1b0_1
      jedi               pkgs/main/linux-64::jedi-0.18.1-py39h06a4308_1
      jinja2             pkgs/main/noarch::jinja2-3.0.2-pyhd3eb1b0_0
      json5              pkgs/main/noarch::json5-0.9.6-pyhd3eb1b0_0
      jsonschema         pkgs/main/noarch::jsonschema-3.2.0-pyhd3eb1b0_2
      jupyter_client     pkgs/main/noarch::jupyter_client-7.1.2-pyhd3eb1b0_0
      jupyter_core       pkgs/main/linux-64::jupyter_core-4.9.1-py39h06a4308_0
      jupyter_server     pkgs/main/linux-64::jupyter_server-1.4.1-py39h06a4308_0
      jupyterlab         pkgs/main/noarch::jupyterlab-3.2.1-pyhd3eb1b0_1
      jupyterlab_pygmen~ pkgs/main/noarch::jupyterlab_pygments-0.1.2-py_0
      jupyterlab_server  pkgs/main/noarch::jupyterlab_server-2.10.2-pyhd3eb1b0_1
      ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.35.1-h7274673_9
      libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
      libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.3.0-h5101ec6_17
      libgomp            pkgs/main/linux-64::libgomp-9.3.0-h5101ec6_17
      libsodium          pkgs/main/linux-64::libsodium-1.0.18-h7b6447c_0
      libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.3.0-hd4cf53a_17
      markupsafe         pkgs/main/linux-64::markupsafe-2.0.1-py39h27cfd23_0
      matplotlib-inline  pkgs/main/noarch::matplotlib-inline-0.1.2-pyhd3eb1b0_2
      mistune            pkgs/main/linux-64::mistune-0.8.4-py39h27cfd23_1000
      nbclassic          pkgs/main/noarch::nbclassic-0.2.6-pyhd3eb1b0_0
      nbclient           pkgs/main/noarch::nbclient-0.5.3-pyhd3eb1b0_0
      nbconvert          pkgs/main/linux-64::nbconvert-6.3.0-py39h06a4308_0
      nbformat           pkgs/main/noarch::nbformat-5.1.3-pyhd3eb1b0_0
      ncurses            pkgs/main/linux-64::ncurses-6.3-h7f8727e_2
      nest-asyncio       pkgs/main/noarch::nest-asyncio-1.5.1-pyhd3eb1b0_0
      notebook           pkgs/main/linux-64::notebook-6.4.6-py39h06a4308_0
      openssl            pkgs/main/linux-64::openssl-1.1.1m-h7f8727e_0
      packaging          pkgs/main/noarch::packaging-21.3-pyhd3eb1b0_0
      pandocfilters      pkgs/main/noarch::pandocfilters-1.5.0-pyhd3eb1b0_0
      parso              pkgs/main/noarch::parso-0.8.3-pyhd3eb1b0_0
      pexpect            pkgs/main/noarch::pexpect-4.8.0-pyhd3eb1b0_3
      pickleshare        pkgs/main/noarch::pickleshare-0.7.5-pyhd3eb1b0_1003
      pip                pkgs/main/linux-64::pip-21.2.4-py39h06a4308_0
      prometheus_client  pkgs/main/noarch::prometheus_client-0.13.1-pyhd3eb1b0_0
      prompt-toolkit     pkgs/main/noarch::prompt-toolkit-3.0.20-pyhd3eb1b0_0
      ptyprocess         pkgs/main/noarch::ptyprocess-0.7.0-pyhd3eb1b0_2
      pycparser          pkgs/main/noarch::pycparser-2.21-pyhd3eb1b0_0
      pygments           pkgs/main/noarch::pygments-2.11.2-pyhd3eb1b0_0
      pyopenssl          pkgs/main/noarch::pyopenssl-22.0.0-pyhd3eb1b0_0
      pyparsing          pkgs/main/noarch::pyparsing-3.0.4-pyhd3eb1b0_0
      pyrsistent         pkgs/main/linux-64::pyrsistent-0.18.0-py39heee7806_0
      pysocks            pkgs/main/linux-64::pysocks-1.7.1-py39h06a4308_0
      python             pkgs/main/linux-64::python-3.9.7-h12debd9_1
      python-dateutil    pkgs/main/noarch::python-dateutil-2.8.2-pyhd3eb1b0_0
      pytz               pkgs/main/noarch::pytz-2021.3-pyhd3eb1b0_0
      pyzmq              pkgs/main/linux-64::pyzmq-22.3.0-py39h295c915_2
      readline           pkgs/main/linux-64::readline-8.1.2-h7f8727e_1
      requests           pkgs/main/noarch::requests-2.27.1-pyhd3eb1b0_0
      send2trash         pkgs/main/noarch::send2trash-1.8.0-pyhd3eb1b0_1
      setuptools         pkgs/main/linux-64::setuptools-58.0.4-py39h06a4308_0
      six                pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_0
      sniffio            pkgs/main/linux-64::sniffio-1.2.0-py39h06a4308_1
      sqlite             pkgs/main/linux-64::sqlite-3.37.2-hc218d9a_0
      terminado          pkgs/main/linux-64::terminado-0.9.4-py39h06a4308_0
      testpath           pkgs/main/noarch::testpath-0.5.0-pyhd3eb1b0_0
      tk                 pkgs/main/linux-64::tk-8.6.11-h1ccaba5_0
      tornado            pkgs/main/linux-64::tornado-6.1-py39h27cfd23_0
      traitlets          pkgs/main/noarch::traitlets-5.1.1-pyhd3eb1b0_0
      tzdata             pkgs/main/noarch::tzdata-2021e-hda174b7_0
      urllib3            pkgs/main/noarch::urllib3-1.26.8-pyhd3eb1b0_0
      wcwidth            pkgs/main/noarch::wcwidth-0.2.5-pyhd3eb1b0_0
      webencodings       pkgs/main/linux-64::webencodings-0.5.1-py39h06a4308_1
      wheel              pkgs/main/noarch::wheel-0.37.1-pyhd3eb1b0_0
      xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
      zeromq             pkgs/main/linux-64::zeromq-4.3.4-h2531618_0
      zipp               pkgs/main/noarch::zipp-3.7.0-pyhd3eb1b0_0
      zlib               pkgs/main/linux-64::zlib-1.2.11-h7f8727e_4
    
    
    Proceed ([y]/n)? y
    
    
    Downloading and Extracting Packages
    send2trash-1.8.0     | 19 KB     | ######################################################################### | 100%
    tk-8.6.11            | 3.0 MB    | ######################################################################### | 100%
    pytz-2021.3          | 171 KB    | ######################################################################### | 100%
    prompt-toolkit-3.0.2 | 259 KB    | ######################################################################### | 100%
    python-3.9.7         | 18.6 MB   | ####################################################7                     |  72%
    python-3.9.7         | 18.6 MB   | ######################################################################### | 100%
    importlib-metadata-4 | 39 KB     | ######################################################################### | 100%
    ipykernel-6.4.1      | 194 KB    | ######################################################################### | 100%
    wcwidth-0.2.5        | 26 KB     | ######################################################################### | 100%
    prometheus_client-0. | 47 KB     | ######################################################################### | 100%
    jupyterlab-3.2.1     | 3.6 MB    | ######################################################################### | 100%
    decorator-5.1.1      | 12 KB     | ######################################################################### | 100%
    packaging-21.3       | 36 KB     | ######################################################################### | 100%
    ca-certificates-2021 | 115 KB    | ######################################################################### | 100%
    zlib-1.2.11          | 108 KB    | ######################################################################### | 100%
    readline-8.1.2       | 354 KB    | ######################################################################### | 100%
    wheel-0.37.1         | 33 KB     | ######################################################################### | 100%
    openssl-1.1.1m       | 2.5 MB    | ######################################################################### | 100%
    ipython-7.31.1       | 1006 KB   | ######################################################################### | 100%
    idna-3.3             | 49 KB     | ######################################################################### | 100%
    tzdata-2021e         | 112 KB    | ######################################################################### | 100%
    urllib3-1.26.8       | 106 KB    | ######################################################################### | 100%
    attrs-21.4.0         | 51 KB     | ######################################################################### | 100%
    jupyter_client-7.1.2 | 93 KB     | ######################################################################### | 100%
    jupyterlab_server-2. | 48 KB     | ######################################################################### | 100%
    zipp-3.7.0           | 12 KB     | ######################################################################### | 100%
    importlib_metadata-4 | 12 KB     | ######################################################################### | 100%
    cffi-1.15.0          | 225 KB    | ######################################################################### | 100%
    traitlets-5.1.1      | 84 KB     | ######################################################################### | 100%
    pyzmq-22.3.0         | 470 KB    | ######################################################################### | 100%
    parso-0.8.3          | 70 KB     | ######################################################################### | 100%
    ncurses-6.3          | 782 KB    | ######################################################################### | 100%
    notebook-6.4.6       | 4.2 MB    | ######################################################################### | 100%
    jinja2-3.0.2         | 110 KB    | ######################################################################### | 100%
    jsonschema-3.2.0     | 47 KB     | ######################################################################### | 100%
    pandocfilters-1.5.0  | 11 KB     | ######################################################################### | 100%
    sqlite-3.37.2        | 1008 KB   | ######################################################################### | 100%
    charset-normalizer-2 | 35 KB     | ######################################################################### | 100%
    pyparsing-3.0.4      | 81 KB     | ######################################################################### | 100%
    pycparser-2.21       | 94 KB     | ######################################################################### | 100%
    pyopenssl-22.0.0     | 50 KB     | ######################################################################### | 100%
    jedi-0.18.1          | 982 KB    | ######################################################################### | 100%
    pyrsistent-0.18.0    | 94 KB     | ######################################################################### | 100%
    matplotlib-inline-0. | 12 KB     | ######################################################################### | 100%
    pip-21.2.4           | 1.8 MB    | ######################################################################### | 100%
    _libgcc_mutex-0.1    | 3 KB      | ######################################################################### | 100%
    debugpy-1.5.1        | 1.7 MB    | ######################################################################### | 100%
    setuptools-58.0.4    | 790 KB    | ######################################################################### | 100%
    certifi-2021.10.8    | 151 KB    | ######################################################################### | 100%
    requests-2.27.1      | 54 KB     | ######################################################################### | 100%
    bleach-4.1.0         | 123 KB    | ######################################################################### | 100%
    nbconvert-6.3.0      | 488 KB    | ######################################################################### | 100%
    jupyter_core-4.9.1   | 75 KB     | ######################################################################### | 100%
    pygments-2.11.2      | 759 KB    | ######################################################################### | 100%
    cryptography-36.0.0  | 1.3 MB    | ######################################################################### | 100%
    Preparing transaction: done
    Verifying transaction: done
    Executing transaction: done
    #
    # To activate this environment, use
    #
    #     $ conda activate tf
    #
    # To deactivate an active environment, use
    #
    #     $ conda deactivate
    
    (base) [ye53nis@login01 drmed-git]$ conda activate tf
    (tf) [ye53nis@login01 drmed-git]$ pip install numpy mlflow lmfit scikit-learn tensorflow matplotlib pandas seaborn f
    csfiles multipletau
    Collecting numpy
      Downloading numpy-1.22.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
         |████████████████████████████████| 16.8 MB 6.6 MB/s
    Collecting mlflow
      Downloading mlflow-1.23.1-py3-none-any.whl (15.6 MB)
         |████████████████████████████████| 15.6 MB 36.5 MB/s
    Collecting lmfit
      Downloading lmfit-1.0.3.tar.gz (292 kB)
         |████████████████████████████████| 292 kB 15.8 MB/s
    Collecting scikit-learn
      Downloading scikit_learn-1.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.4 MB)
         |████████████████████████████████| 26.4 MB 35.6 MB/s
    Collecting tensorflow
      Downloading tensorflow-2.8.0-cp39-cp39-manylinux2010_x86_64.whl (497.6 MB)
         |████████████████████████████████| 497.6 MB 15 kB/s
    Collecting matplotlib
      Downloading matplotlib-3.5.1-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (11.2 MB)
         |████████████████████████████████| 11.2 MB 34.3 MB/s
    Collecting pandas
      Downloading pandas-1.4.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.7 MB)
         |████████████████████████████████| 11.7 MB 35.3 MB/s
    Collecting seaborn
      Downloading seaborn-0.11.2-py3-none-any.whl (292 kB)
         |████████████████████████████████| 292 kB 36.2 MB/s
    Collecting fcsfiles
      Downloading fcsfiles-2022.2.2-py3-none-any.whl (9.6 kB)
    Collecting multipletau
      Using cached multipletau-0.3.3-py2.py3-none-any.whl (12 kB)
    Collecting sqlparse>=0.3.1
      Downloading sqlparse-0.4.2-py3-none-any.whl (42 kB)
         |████████████████████████████████| 42 kB 1.3 MB/s
    Collecting querystring-parser
      Using cached querystring_parser-1.2.4-py2.py3-none-any.whl (7.9 kB)
    Collecting alembic
      Downloading alembic-1.7.6-py3-none-any.whl (210 kB)
         |████████████████████████████████| 210 kB 41.2 MB/s
    Collecting databricks-cli>=0.8.7
      Downloading databricks-cli-0.16.4.tar.gz (58 kB)
         |████████████████████████████████| 58 kB 7.0 MB/s
    Requirement already satisfied: importlib-metadata!=4.7.0,>=3.7.0 in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-
    packages (from mlflow) (4.8.2)
    Requirement already satisfied: pytz in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (from mlflow) (2021.
    3)
    Collecting sqlalchemy
      Downloading SQLAlchemy-1.4.31-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014
    _x86_64.whl (1.6 MB)
         |████████████████████████████████| 1.6 MB 33.9 MB/s
    Requirement already satisfied: entrypoints in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (from mlflow)
     (0.3)
    Requirement already satisfied: requests>=2.17.3 in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (from ml
    flow) (2.27.1)
    Collecting Flask
      Downloading Flask-2.0.2-py3-none-any.whl (95 kB)
         |████████████████████████████████| 95 kB 4.8 MB/s
    Collecting gitpython>=2.1.0
      Downloading GitPython-3.1.26-py3-none-any.whl (180 kB)
         |████████████████████████████████| 180 kB 43.3 MB/s
    Collecting click>=7.0
      Downloading click-8.0.3-py3-none-any.whl (97 kB)
         |████████████████████████████████| 97 kB 7.1 MB/s
    Collecting protobuf>=3.7.0
      Downloading protobuf-3.19.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
         |████████████████████████████████| 1.1 MB 35.5 MB/s
    Collecting docker>=4.0.0
      Downloading docker-5.0.3-py2.py3-none-any.whl (146 kB)
         |████████████████████████████████| 146 kB 43.2 MB/s
    Collecting pyyaml>=5.1
      Downloading PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64
    .whl (661 kB)
         |████████████████████████████████| 661 kB 36.9 MB/s
    Collecting scipy
      Downloading scipy-1.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (42.1 MB)
         |████████████████████████████████| 42.1 MB 33.3 MB/s
    Collecting cloudpickle
      Downloading cloudpickle-2.0.0-py3-none-any.whl (25 kB)
    Requirement already satisfied: packaging in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (from mlflow) (
    21.3)
    Collecting gunicorn
      Using cached gunicorn-20.1.0-py3-none-any.whl (79 kB)
    Collecting prometheus-flask-exporter
      Downloading prometheus_flask_exporter-0.18.7-py3-none-any.whl (17 kB)
    Collecting asteval>=0.9.22
      Downloading asteval-0.9.26.tar.gz (40 kB)
         |████████████████████████████████| 40 kB 4.2 MB/s
    Collecting uncertainties>=3.0.1
      Using cached uncertainties-3.1.6-py2.py3-none-any.whl (98 kB)
    Collecting joblib>=0.11
      Downloading joblib-1.1.0-py2.py3-none-any.whl (306 kB)
         |████████████████████████████████| 306 kB 37.4 MB/s
    Collecting threadpoolctl>=2.0.0
      Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
    Requirement already satisfied: setuptools in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (from tensorfl
    ow) (58.0.4)
    Collecting keras-preprocessing>=1.1.1
      Using cached Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
    Collecting google-pasta>=0.1.1
      Using cached google_pasta-0.2.0-py3-none-any.whl (57 kB)
    Collecting termcolor>=1.1.0
      Using cached termcolor-1.1.0-py3-none-any.whl
    Collecting tensorflow-io-gcs-filesystem>=0.23.1
      Downloading tensorflow_io_gcs_filesystem-0.24.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.1 MB)
         |████████████████████████████████| 2.1 MB 35.6 MB/s
    Collecting gast>=0.2.1
      Downloading gast-0.5.3-py3-none-any.whl (19 kB)
    Collecting wrapt>=1.11.0
      Downloading wrapt-1.13.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_
    64.whl (81 kB)
         |████████████████████████████████| 81 kB 11.3 MB/s
    Collecting h5py>=2.9.0
      Downloading h5py-3.6.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.5 MB)
         |████████████████████████████████| 4.5 MB 41.8 MB/s
    Requirement already satisfied: six>=1.12.0 in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (from tensorf
    low) (1.16.0)
    Collecting opt-einsum>=2.3.2
      Using cached opt_einsum-3.3.0-py3-none-any.whl (65 kB)
    Collecting tf-estimator-nightly==2.8.0.dev2021122109
      Downloading tf_estimator_nightly-2.8.0.dev2021122109-py2.py3-none-any.whl (462 kB)
         |████████████████████████████████| 462 kB 37.0 MB/s
    Collecting typing-extensions>=3.6.6
      Downloading typing_extensions-4.0.1-py3-none-any.whl (22 kB)
    Collecting tensorboard<2.9,>=2.8
      Downloading tensorboard-2.8.0-py3-none-any.whl (5.8 MB)
         |████████████████████████████████| 5.8 MB 35.0 MB/s
    Collecting grpcio<2.0,>=1.24.3
      Downloading grpcio-1.43.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.1 MB)
         |████████████████████████████████| 4.1 MB 34.1 MB/s
    Collecting astunparse>=1.6.0
      Using cached astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
    Collecting flatbuffers>=1.12
      Downloading flatbuffers-2.0-py2.py3-none-any.whl (26 kB)
    Collecting libclang>=9.0.1
      Downloading libclang-13.0.0-py2.py3-none-manylinux1_x86_64.whl (14.5 MB)
         |████████████████████████████████| 14.5 MB 31.0 MB/s
    Collecting absl-py>=0.4.0
      Downloading absl_py-1.0.0-py3-none-any.whl (126 kB)
         |████████████████████████████████| 126 kB 38.1 MB/s
    Collecting keras<2.9,>=2.8.0rc0
      Downloading keras-2.8.0-py2.py3-none-any.whl (1.4 MB)
         |████████████████████████████████| 1.4 MB 37.0 MB/s
    Collecting pillow>=6.2.0
      Downloading Pillow-9.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
         |████████████████████████████████| 4.3 MB 38.4 MB/s
    Requirement already satisfied: python-dateutil>=2.7 in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (fro
    m matplotlib) (2.8.2)
    Collecting fonttools>=4.22.0
      Downloading fonttools-4.29.1-py3-none-any.whl (895 kB)
         |████████████████████████████████| 895 kB 45.3 MB/s
    Requirement already satisfied: pyparsing>=2.2.1 in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (from ma
    tplotlib) (3.0.4)
    Collecting kiwisolver>=1.0.1
      Downloading kiwisolver-1.3.2-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.6 MB)
         |████████████████████████████████| 1.6 MB 33.7 MB/s
    Collecting cycler>=0.10
      Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
    Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (from
    astunparse>=1.6.0->tensorflow) (0.37.1)
    Collecting tabulate>=0.7.7
      Using cached tabulate-0.8.9-py3-none-any.whl (25 kB)
    Collecting websocket-client>=0.32.0
      Downloading websocket_client-1.2.3-py3-none-any.whl (53 kB)
         |████████████████████████████████| 53 kB 1.5 MB/s
    Collecting gitdb<5,>=4.0.1
      Downloading gitdb-4.0.9-py3-none-any.whl (63 kB)
         |████████████████████████████████| 63 kB 1.4 MB/s
    Collecting smmap<6,>=3.0.1
      Downloading smmap-5.0.0-py3-none-any.whl (24 kB)
    Requirement already satisfied: zipp>=0.5 in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (from importlib
    -metadata!=4.7.0,>=3.7.0->mlflow) (3.7.0)
    Requirement already satisfied: idna<4,>=2.5 in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (from reques
    ts>=2.17.3->mlflow) (3.3)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (fr
    om requests>=2.17.3->mlflow) (1.26.8)
    Requirement already satisfied: certifi>=2017.4.17 in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (from
    requests>=2.17.3->mlflow) (2021.10.8)
    Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages
     (from requests>=2.17.3->mlflow) (2.0.4)
    Collecting tensorboard-data-server<0.7.0,>=0.6.0
      Using cached tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
    Collecting werkzeug>=0.11.15
      Downloading Werkzeug-2.0.3-py3-none-any.whl (289 kB)
         |████████████████████████████████| 289 kB 42.0 MB/s
    Collecting google-auth<3,>=1.6.3
      Downloading google_auth-2.6.0-py2.py3-none-any.whl (156 kB)
         |████████████████████████████████| 156 kB 46.0 MB/s
    Collecting tensorboard-plugin-wit>=1.6.0
      Downloading tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
         |████████████████████████████████| 781 kB 34.7 MB/s
    Collecting markdown>=2.6.8
      Downloading Markdown-3.3.6-py3-none-any.whl (97 kB)
         |████████████████████████████████| 97 kB 6.1 MB/s
    Collecting google-auth-oauthlib<0.5,>=0.4.1
      Downloading google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
    Collecting cachetools<6.0,>=2.0.0
      Downloading cachetools-5.0.0-py3-none-any.whl (9.1 kB)
    Collecting rsa<5,>=3.1.4
      Downloading rsa-4.8-py3-none-any.whl (39 kB)
    Collecting pyasn1-modules>=0.2.1
      Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
    Collecting requests-oauthlib>=0.7.0
      Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
    Collecting pyasn1<0.5.0,>=0.4.6
      Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
    Collecting oauthlib>=3.0.0
      Downloading oauthlib-3.2.0-py3-none-any.whl (151 kB)
         |████████████████████████████████| 151 kB 41.0 MB/s
    Collecting future
      Using cached future-0.18.2-py3-none-any.whl
    Collecting Mako
      Downloading Mako-1.1.6-py2.py3-none-any.whl (75 kB)
         |████████████████████████████████| 75 kB 3.8 MB/s
    Collecting greenlet!=0.4.17
      Downloading greenlet-1.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (153 kB)
         |████████████████████████████████| 153 kB 44.8 MB/s
    Requirement already satisfied: Jinja2>=3.0 in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (from Flask->
    mlflow) (3.0.2)
    Collecting itsdangerous>=2.0
      Using cached itsdangerous-2.0.1-py3-none-any.whl (18 kB)
    Requirement already satisfied: MarkupSafe>=2.0 in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (from Jin
    ja2>=3.0->Flask->mlflow) (2.0.1)
    Requirement already satisfied: prometheus-client in /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages (from p
    rometheus-flask-exporter->mlflow) (0.13.1)
    Building wheels for collected packages: lmfit, asteval, databricks-cli
      Building wheel for lmfit (setup.py) ... done
      Created wheel for lmfit: filename=lmfit-1.0.3-py3-none-any.whl size=84402 sha256=eeec89959143b4c2a2cb11c2a3e94314e
    fe83e19494f90b784e75ec988670aad
      Stored in directory: /home/ye53nis/.cache/pip/wheels/76/f4/32/c336957bfd694c7746f4df19b74e08d918ada688fe1349cca2
      Building wheel for asteval (setup.py) ... done
      Created wheel for asteval: filename=asteval-0.9.26-py3-none-any.whl size=17648 sha256=cf0f0e455c27d314babbe54d3e60
    886323c9dc39f51f6e6f9024972a201ff9d1
      Stored in directory: /home/ye53nis/.cache/pip/wheels/2f/e1/8d/9c9d29d91b8e6e79c0de5d06a09b5e69b0e4e390fa9765a449
      Building wheel for databricks-cli (setup.py) ... done
      Created wheel for databricks-cli: filename=databricks_cli-0.16.4-py3-none-any.whl size=106877 sha256=71c0798975bca
    4589dde21e027bf35118ae80567838d55515827123c6b3bb93a
      Stored in directory: /home/ye53nis/.cache/pip/wheels/e2/12/4a/3df4a44571c7f53c075a50e850162af541ea24b2a689d517ac
    Successfully built lmfit asteval databricks-cli
    Installing collected packages: pyasn1, rsa, pyasn1-modules, oauthlib, cachetools, werkzeug, smmap, requests-oauthlib
    , itsdangerous, greenlet, google-auth, click, websocket-client, tensorboard-plugin-wit, tensorboard-data-server, tab
    ulate, sqlalchemy, protobuf, pillow, numpy, markdown, Mako, kiwisolver, grpcio, google-auth-oauthlib, gitdb, future,
     fonttools, Flask, cycler, absl-py, wrapt, uncertainties, typing-extensions, threadpoolctl, tf-estimator-nightly, te
    rmcolor, tensorflow-io-gcs-filesystem, tensorboard, sqlparse, scipy, querystring-parser, pyyaml, prometheus-flask-ex
    porter, pandas, opt-einsum, matplotlib, libclang, keras-preprocessing, keras, joblib, h5py, gunicorn, google-pasta,
    gitpython, gast, flatbuffers, docker, databricks-cli, cloudpickle, astunparse, asteval, alembic, tensorflow, seaborn
    , scikit-learn, multipletau, mlflow, lmfit, fcsfiles
    Successfully installed Flask-2.0.2 Mako-1.1.6 absl-py-1.0.0 alembic-1.7.6 asteval-0.9.26 astunparse-1.6.3 cachetools
    -5.0.0 click-8.0.3 cloudpickle-2.0.0 cycler-0.11.0 databricks-cli-0.16.4 docker-5.0.3 fcsfiles-2022.2.2 flatbuffers-
    2.0 fonttools-4.29.1 future-0.18.2 gast-0.5.3 gitdb-4.0.9 gitpython-3.1.26 google-auth-2.6.0 google-auth-oauthlib-0.
    4.6 google-pasta-0.2.0 greenlet-1.1.2 grpcio-1.43.0 gunicorn-20.1.0 h5py-3.6.0 itsdangerous-2.0.1 joblib-1.1.0 keras
    -2.8.0 keras-preprocessing-1.1.2 kiwisolver-1.3.2 libclang-13.0.0 lmfit-1.0.3 markdown-3.3.6 matplotlib-3.5.1 mlflow
    -1.23.1 multipletau-0.3.3 numpy-1.22.2 oauthlib-3.2.0 opt-einsum-3.3.0 pandas-1.4.0 pillow-9.0.1 prometheus-flask-ex
    porter-0.18.7 protobuf-3.19.4 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyyaml-6.0 querystring-parser-1.2.4 requests-oauthli
    b-1.3.1 rsa-4.8 scikit-learn-1.0.2 scipy-1.8.0 seaborn-0.11.2 smmap-5.0.0 sqlalchemy-1.4.31 sqlparse-0.4.2 tabulate-
    0.8.9 tensorboard-2.8.0 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-2.8.0 tensorflow-io-gc
    s-filesystem-0.24.0 termcolor-1.1.0 tf-estimator-nightly-2.8.0.dev2021122109 threadpoolctl-3.1.0 typing-extensions-4
    .0.1 uncertainties-3.1.6 websocket-client-1.2.3 werkzeug-2.0.3 wrapt-1.13.3
    (tf) [ye53nis@login01 applications]$ pip install cython --no-binary cython
    Collecting cython
      Downloading Cython-0.29.27.tar.gz (2.1 MB)
         |████████████████████████████████| 2.1 MB 6.7 MB/s
    Skipping wheel build for cython, due to binaries being disabled for it.
    Installing collected packages: cython
        Running setup.py install for cython ... done
    Successfully installed cython-0.29.27
    
  4. Now Install dividAndConquer as a Cython implementation
    cd /beegfs/ye53nis/drmed-git/src/fluotracify/applications
    cat _correlate_cython_setup.py
    python _correlate_cython_setup.py build_ext --inplace
    
    (tf) [ye53nis@login01 applications]$ cd /beegfs/ye53nis/drmed-git/src/fluotracify/applications
    (tf) [ye53nis@login01 applications]$ cat _correlate_cython_setup.py
    import setuptools
    from Cython.Build import cythonize
    import numpy as np
    
    
    setuptools.setup(
        ext_modules=cythonize("correlate_cython.pyx",
                              compiler_directives={'language_level': '3'}),
        include_dirs=[np.get_include()]
    )
    (tf) [ye53nis@login01 applications]$ python _correlate_cython_setup.py build_ext --inplace
    running build_ext
    copying build/lib.linux-x86_64-3.9/correlate_cython.cpython-39-x86_64-linux-gnu.so ->
    (tf) [ye53nis@login01 applications]$
    
  5. Load all needed modules
    import logging
    import os
    import sys
    import tracemalloc
    
    from pathlib import Path
    from pprint import pprint
    from tensorflow.keras.optimizers import Adam
    from mlflow.keras import load_model
    
    FLUOTRACIFY_PATH = '/beegfs/ye53nis/drmed-git/src/'
    sys.path.append(FLUOTRACIFY_PATH)
    from fluotracify.applications import corr_fit_object as cfo
    from fluotracify.training import build_model as bm
    
    logging.basicConfig(filename='data/exp-220120-correlate-ptu/2022-02-08_correlate-all-clean-ptu.log',
                        filemode='w', format='%(asctime)s - %(message)s',
                        force=True,
                        level=logging.DEBUG)
    
    
    2022-02-10 01:15:43.026025: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2022-02-10 01:15:43.026059: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    
  6. Define variables and prepare model. Since the first run I changed the folder names assigned to path, because I noticed that I mixed up the experiments I wanted to analyse: the "
    class ParameterClass():
        """Stores parameters for correlation """
        def __init__(self):
            # Where the data is stored.
            self.data = []
            self.objectRef = []
            self.subObjectRef = []
            self.colors = ['blue', 'green', 'red', 'cyan', 'magenta',
                           'yellow', 'black']
            self.numOfLoaded = 0
            # very fast from Ncasc ~ 14 onwards
            self.NcascStart = 0
            self.NcascEnd = 30  # 25
            self.Nsub = 6  # 6
            self.photonLifetimeBin = 10  # used for photon decay
            self.photonCountBin = 1  # used for time series
    
    path = Path("../data/1911DD_atto+LUVs/all_clean_ptu")
    logged_model = 'file:///beegfs/ye53nis/drmed-git/data/mlruns/8/00f2635d9fa2463c9a066722163405be/artifacts/model'
    mt_bin = 0.001
    output_path = '/beegfs/ye53nis/saves/2022-02-08_correlate-all-clean-ptu/'
    
    par_obj = ParameterClass()
    
    loaded_model = load_model(logged_model, compile=False)
    loaded_model.compile(loss=bm.binary_ce_dice_loss(),
                         optimizer=Adam(),
                         metrics = bm.unet_metrics([0.1, 0.3, 0.5, 0.7, 0.9]))
    bm.prepare_model(loaded_model, [2**14, 2**13, 2**12, 2**15, 2**13])
    
    
    2022-02-10 01:15:50.033533: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2022-02-10 01:15:50.033566: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
    2022-02-10 01:15:50.033596: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node154): /proc/driver/nvidia/version does not exist
    2022-02-10 01:15:50.033828: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    
  7. Run experiment: Correlate all_clean_ptu of the AlexaFluor488 experiment on Jupyter 1. We want the following correlations:
    • clean trace → correlation via tttr2xfcs, and multipletau (bin=0.001)
    • correction=delete → correlation via tttr2xfcs, and multipletau (bin=0.001)
    • correction=delete_and_shift → correlation via tttr2xfcs, and multipletau (bin=0.001)
    • correction=weights-random → correlation via tttr2xfcs
    • correction=weights-1-pred → correlation via tttr2xfcs
      mt_bin = 0.001
      def run_correlations(myfile):
          ptufile = cfo.PicoObject(myfile, par_obj)
          ptufile.getTimeSeries(photonCountBin=mt_bin,
                                timeseries_name=f'us_{ptufile.name}_NOCORR')
          ptufile.getPhotonCountingStats(name=f'us_{ptufile.name}_NOCORR')
          for method in ['delete', 'delete_and_shift']:
              mt_name = f'us_{ptufile.name}_{method}'
              ptufile.getTimeSeries(timeseries_name=mt_name)
              ptufile.getPhotonCountingStats(name=mt_name)
              ptufile.predictTimeSeries(model=loaded_model,
                                        scaler='minmax',
                                        name=mt_name)
              ptufile.correctTCSPC(method=method,
                                   bin_after_correction=mt_bin,
                                   timeseries_name=mt_name)
          for weight in ['random', '1-pred']:
              weight_name = f'{ptufile.name}_weights_{weight}'
              ptufile.getTimeSeries(photonCountBin=1.0,
                                    timeseries_name=weight_name)
              ptufile.getPhotonCountingStats(name=weight_name)
              ptufile.predictTimeSeries(model=loaded_model,
                                        scaler='minmax',
                                        name=weight_name)
              ptufile.correctTCSPC(method='weights',
                                   weight=weight,
                                   timeseries_name=weight_name)
          for key in list(ptufile.trueTimeArr.keys()):
              if "_FORWEIGHTS" in key:
                  ptufile.get_autocorrelation(method='tttr2xfcs_with_weights',
                                              name=key)
              else:
                  ptufile.get_autocorrelation(method='tttr2xfcs', name=key)
          for key in list(ptufile.timeSeries.keys()):
              for k, i in ptufile.timeSeries[key].items():
                  if key.endswith(('_DELBIN', '_DELSHIFTBIN', '_NOCORR')):
                      ptufile.get_autocorrelation(method='multipletau', name=(key, k))
      
          for m in ['multipletau', 'tttr2xfcs', 'tttr2xfcs_with_weights']:
              if m in list(ptufile.autoNorm.keys()):
                  for key in list(ptufile.autoNorm[m].keys()):
                      ptufile.save_autocorrelation(name=key, method=m,
                                                   output_path=output_path)
          del ptufile
      
      def sizeof_fmt(num, suffix='B'):
          ''' by Fred Cirera,  https://stackoverflow.com/a/1094933/1870254, modified'''
          for unit in ['','Ki','Mi','Gi','Ti','Pi','Ei','Zi']:
              if abs(num) < 1024.0:
                  return "%3.1f %s%s" % (num, unit, suffix)
              num /= 1024.0
          return "%.1f %s%s" % (num, 'Yi', suffix)
      
      files = [path / f for f in os.listdir(path) if f.endswith('.ptu')]
      
      if len(files) == 0:
          raise FileNotFoundError('The path provided does not include any'
                                  ' .ptu files.')
      tracemalloc.start()
      tracemalloc_dict = {}
      tracemalloc_dict['baseline'] = tracemalloc.take_snapshot()
      for i, myfile in enumerate(files):
          if i < 2:
              print(i)
              run_correlations(myfile)
              tracemalloc_dict[f'{i}'] = tracemalloc.take_snapshot()
      
      0
      /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      1
      /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      
  8. According to log, the first run ended with an Error
    [I 2022-01-22 21:32:42.402 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), keep random ports
    

    Only 229 traces were analyzed.

    tracemalloc_dict.keys()
    
    dict_keys(['baseline', '0', '1'])
    
    top_stats_base = tracemalloc_dict['baseline'].statistics("lineno")
    top_stats_0 = tracemalloc_dict['0'].statistics("lineno")
    top_stats_1 = tracemalloc_dict['1'].statistics("lineno")
    top_stats_comp = tracemalloc_dict['1'].compare_to(tracemalloc_dict['0'], "lineno")
    print("---------------------------------------------------------")
    
    ---------------------------------------------------------
    
    [print(stat) for stat in top_stats_base[:10]]
    print("---------------------------------------------------------")
    [print(stat) for stat in top_stats_0[:10]]
    print("---------------------------------------------------------")
    [print(stat) for stat in top_stats_comp[:10]]
    
    /home/ye53nis/.conda/envs/tf/lib/python3.9/codeop.py:143: size=225 B, count=2, average=112 B
    /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/IPython/core/interactiveshell.py:3377: size=120 B, count=1, average=120 B
    /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/IPython/core/interactiveshell.py:3371: size=96 B, count=3, average=32 B
    /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/IPython/core/interactiveshell.py:3441: size=64 B, count=1, average=64 B
    /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/IPython/core/compilerop.py:178: size=28 B, count=1, average=28 B
    ---------------------------------------------------------
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:424: size=431 MiB, count=34, average=12.7 MiB
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:732: size=107 MiB, count=4, average=26.6 MiB
    /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/numpy/lib/function_base.py:5131: size=87.5 MiB, count=10, average=8962 KiB
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:662: size=53.3 MiB, count=8, average=6817 KiB
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:134: size=26.6 MiB, count=2, average=13.3 MiB
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:132: size=26.6 MiB, count=2, average=13.3 MiB
    /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/tensorflow/python/framework/ops.py:500: size=353 KiB, count=2037, average=177 B
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:550: size=257 KiB, count=9, average=28.6 KiB
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:556: size=257 KiB, count=12, average=21.4 KiB
    /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/tensorflow/python/framework/ops.py:2110: size=235 KiB, count=2072, average=116 B
    ---------------------------------------------------------
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:424: size=855 MiB (+424 MiB), count=68 (+34), average=12.6 MiB
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:732: size=213 MiB (+107 MiB), count=8 (+4), average=26.7 MiB
    /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/numpy/lib/function_base.py:5131: size=171 MiB (+83.1 MiB), count=20 (+10), average=8737 KiB
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:662: size=107 MiB (+53.4 MiB), count=16 (+8), average=6828 KiB
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:134: size=53.3 MiB (+26.7 MiB), count=4 (+2), average=13.3 MiB
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:132: size=53.3 MiB (+26.7 MiB), count=4 (+2), average=13.3 MiB
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:556: size=513 KiB (+257 KiB), count=24 (+12), average=21.4 KiB
    /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:550: size=514 KiB (+256 KiB), count=17 (+8), average=30.2 KiB
    /home/ye53nis/.conda/envs/tf/lib/python3.9/tracemalloc.py:558: size=83.3 KiB (+83.2 KiB), count=1533 (+1532), average=56 B
    /beegfs/ye53nis/drmed-git/src/fluotracify/imports/ptu_utils.py:324: size=105 KiB (+28.2 KiB), count=1923 (+516), average=56 B
    
    import linecache
    
    def display_top(snapshot, key_type='lineno', limit=10):
        snapshot = snapshot.filter_traces((
            tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
            tracemalloc.Filter(False, "<unknown>"),
        ))
        top_stats = snapshot.statistics(key_type)
    
        print("Top %s lines" % limit)
        for index, stat in enumerate(top_stats[:limit], 1):
            frame = stat.traceback[0]
            print("#%s: %s:%s: %.1f KiB"
                  % (index, frame.filename, frame.lineno, stat.size / 1024))
            line = linecache.getline(frame.filename, frame.lineno).strip()
            if line:
                print('    %s' % line)
    
        other = top_stats[limit:]
        if other:
            size = sum(stat.size for stat in other)
            print("%s other: %.1f KiB" % (len(other), size / 1024))
        total = sum(stat.size for stat in top_stats)
        print("Total allocated size: %.1f KiB" % (total / 1024))
    
    display_top(tracemalloc_dict['1'])
    
    Top 10 lines
    #1: /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:732: 218468.5 KiB
        photon_weights = np.zeros((subChanCorrected.shape[0], 2))
    #2: /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/numpy/lib/function_base.py:5131: 174748.2 KiB
        new = arr[tuple(slobj)]
    #3: /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:662: 109240.8 KiB
        trueTimeCorrected = trueTimeCorrected[channelMask]
    #4: /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:134: 54620.3 KiB
        trueTimeArr = np.array([i for i in trueTimeArrFull
    #5: /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:132: 54620.3 KiB
        subChanArr = np.array([i for i in subChanArrFull
    #6: /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:424: 2441.8 KiB
        return np.array(photons_in_bin), np.array(bins_scale)
    #7: /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:550: 513.7 KiB
        predictions = model.predict(trace, verbose=0).flatten()
    #8: /beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py:556: 513.3 KiB
        self.timeSeries[name][f'{key}_PREPRO'] = trace.flatten().astype(
    #9: /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/tensorflow/python/framework/ops.py:500: 352.8 KiB
        self._consumers = []
    #10: /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/tensorflow/python/framework/ops.py:2110: 234.7 KiB
        self._graph = g
    606 other: 1929.5 KiB
    Total allocated size: 617683.8 KiB
    
    mt_bin = 0.001
    def run_correlations(myfile):
        ptufile = cfo.PicoObject(myfile, par_obj)
        ptufile.getTimeSeries(photonCountBin=mt_bin,
                              timeseries_name=f'us_{ptufile.name}_NOCORR')
        ptufile.getPhotonCountingStats(name=f'us_{ptufile.name}_NOCORR')
        for method in ['delete', 'delete_and_shift']:
            mt_name = f'us_{ptufile.name}_{method}'
            ptufile.getTimeSeries(timeseries_name=mt_name)
            ptufile.getPhotonCountingStats(name=mt_name)
            ptufile.predictTimeSeries(model=loaded_model,
                                      scaler='minmax',
                                      name=mt_name)
            ptufile.correctTCSPC(method=method,
                                 bin_after_correction=mt_bin,
                                 timeseries_name=mt_name)
        for weight in ['random', '1-pred']:
            weight_name = f'{ptufile.name}_weights_{weight}'
            ptufile.getTimeSeries(photonCountBin=1.0,
                                  timeseries_name=weight_name)
            ptufile.getPhotonCountingStats(name=weight_name)
            ptufile.predictTimeSeries(model=loaded_model,
                                      scaler='minmax',
                                      name=weight_name)
            ptufile.correctTCSPC(method='weights',
                                 weight=weight,
                                 timeseries_name=weight_name)
        for key in list(ptufile.trueTimeArr.keys()):
            if "_FORWEIGHTS" in key:
                ptufile.get_autocorrelation(method='tttr2xfcs_with_weights',
                                            name=key)
            else:
                ptufile.get_autocorrelation(method='tttr2xfcs', name=key)
        for key in list(ptufile.timeSeries.keys()):
            for k, i in ptufile.timeSeries[key].items():
                if key.endswith(('_DELBIN', '_DELSHIFTBIN', '_NOCORR')):
                    ptufile.get_autocorrelation(method='multipletau', name=(key, k))
    
        for m in ['multipletau', 'tttr2xfcs', 'tttr2xfcs_with_weights']:
            if m in list(ptufile.autoNorm.keys()):
                for key in list(ptufile.autoNorm[m].keys()):
                    ptufile.save_autocorrelation(name=key, method=m,
                                                 output_path=output_path)
        del ptufile
    
    files = [path / f for f in os.listdir(path) if f.endswith('.ptu')]
    
    if len(files) == 0:
        raise FileNotFoundError('The path provided does not include any'
                                ' .ptu files.')
    
    for i, myfile in enumerate(files):
        run_correlations(myfile)
    
    /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
      warnings.warn("Input dtype is not float; casting to np.float_!",
    

2.5.9 Experiment 2b: Correlate all_dirty_ptu on Jupyter 2

  1. Make sure we are in the correct directory, also do a git log -3 to document latest git commits
    %cd /beegfs/ye53nis/drmed-git
    
    !git log -3
    
    /beegfs/ye53nis/drmed-git
    commit dffd95906e760976dfdfd635b50cf056c8813ca3
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Wed Feb 9 23:24:34 2022 +0100
    
        Fix memory leak by deleting variables 2
    
    commit 298240ae97e683133071500ff83e0dbd2434883e
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Wed Feb 9 23:20:32 2022 +0100
    
        Fix memory leak by deleting variables
    
    commit b2987fdc6615e71118ce28f47bc5174109c72ff1
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Tue Feb 8 17:24:12 2022 +0100
    
        Fix get_autocorrelation tttr2xfcs_with_weights
    
  2. Create directory where we want to save our correlations in
    %mkdir /beegfs/ye53nis/saves/2022-02-08_correlate-all-dirty-ptu/
    
  3. Installing dividAndConquer as a Cython implementation was already done in Experiment 2a
  4. Load all needed modules
    import logging
    import os
    import sys
    
    from pathlib import Path
    from pprint import pprint
    from tensorflow.keras.optimizers import Adam
    from mlflow.keras import load_model
    
    FLUOTRACIFY_PATH = '/beegfs/ye53nis/drmed-git/src/'
    sys.path.append(FLUOTRACIFY_PATH)
    from fluotracify.applications import corr_fit_object as cfo
    from fluotracify.training import build_model as bm
    
    logging.basicConfig(filename='data/exp-220120-correlate-ptu/2022-02-08_correlate-all-dirty-ptu.log',
                        filemode='w', format='%(asctime)s - %(message)s',
                        force=True,
                        level=logging.DEBUG)
    
    
    2022-02-10 01:16:31.273603: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2022-02-10 01:16:31.273668: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    
  5. Define variables and prepare model
    class ParameterClass():
        """Stores parameters for correlation """
        def __init__(self):
            # Where the data is stored.
            self.data = []
            self.objectRef = []
            self.subObjectRef = []
            self.colors = ['blue', 'green', 'red', 'cyan', 'magenta',
                           'yellow', 'black']
            self.numOfLoaded = 0
            # very fast from Ncasc ~ 14 onwards
            self.NcascStart = 0
            self.NcascEnd = 30  # 25
            self.Nsub = 6  # 6
            self.photonLifetimeBin = 10  # used for photon decay
            self.photonCountBin = 1  # used for time series
    
    path = Path("../data/1911DD_atto+LUVs/all_dirty_ptu")
    logged_model = 'file:///beegfs/ye53nis/drmed-git/data/mlruns/8/00f2635d9fa2463c9a066722163405be/artifacts/model'
    mt_bin = 0.001
    output_path = '/beegfs/ye53nis/saves/2022-02-08_correlate-all-dirty-ptu/'
    
    par_obj = ParameterClass()
    
    loaded_model = load_model(logged_model, compile=False)
    loaded_model.compile(loss=bm.binary_ce_dice_loss(),
                         optimizer=Adam(),
                         metrics = bm.unet_metrics([0.1, 0.3, 0.5, 0.7, 0.9]))
    bm.prepare_model(loaded_model, [2**14, 2**13, 2**12, 2**15, 2**13])
    
    
    2022-02-10 01:16:39.134361: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2022-02-10 01:16:39.134392: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
    2022-02-10 01:16:39.134429: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node169): /proc/driver/nvidia/version does not exist
    2022-02-10 01:16:39.134678: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    
  6. Run experiment: Correlate all_dirty_ptu of the AlexaFluor488 + DiO LUVs experiment on Jupyter 2. We want the following correlations:
    • dirty trace → correlation via tttr2xfcs, and multipletau (bin=0.001)
    • correction=delete → correlation via tttr2xfcs, and multipletau (bin=0.001)
    • correction=delete_and_shift → correlation via tttr2xfcs, and multipletau (bin=0.001)
    • correction=weights-random → correlation via tttr2xfcs
    • correction=weights-1-pred → correlation via tttr2xfcs
      files = [path / f for f in os.listdir(path) if f.endswith('.ptu')]
      mt_bin = 0.001
      
      if len(files) == 0:
          raise FileNotFoundError('The path provided does not include any'
                                  ' .ptu files.')
      for myfile in files:
          ptufile = cfo.PicoObject(myfile, par_obj)
          ptufile.getTimeSeries(photonCountBin=mt_bin,
                                timeseries_name=f'us_{ptufile.name}_NOCORR')
          ptufile.getPhotonCountingStats(name=f'us_{ptufile.name}_NOCORR')
          for method in ['delete', 'delete_and_shift']:
              mt_name = f'us_{ptufile.name}_{method}'
              ptufile.getTimeSeries(timeseries_name=mt_name)
              ptufile.getPhotonCountingStats(name=mt_name)
              ptufile.predictTimeSeries(model=loaded_model,
                                        scaler='minmax',
                                        name=mt_name)
              ptufile.correctTCSPC(method=method,
                                   bin_after_correction=mt_bin,
                                   timeseries_name=mt_name)
          for weight in ['random', '1-pred']:
              weight_name = f'{ptufile.name}_weights_{weight}'
              ptufile.getTimeSeries(photonCountBin=1.0,
                                    timeseries_name=weight_name)
              ptufile.getPhotonCountingStats(name=weight_name)
              ptufile.predictTimeSeries(model=loaded_model,
                                        scaler='minmax',
                                        name=weight_name)
              ptufile.correctTCSPC(method='weights',
                                   weight=weight,
                                   timeseries_name=weight_name)
          for key in list(ptufile.trueTimeArr.keys()):
              if "_FORWEIGHTS" in key:
                  ptufile.get_autocorrelation(method='tttr2xfcs_with_weights',
                                              name=key)
              else:
                  ptufile.get_autocorrelation(method='tttr2xfcs', name=key)
          for key in list(ptufile.timeSeries.keys()):
              for k, i in ptufile.timeSeries[key].items():
                  if key.endswith(('_DELBIN', '_DELSHIFTBIN', '_NOCORR')):
                      ptufile.get_autocorrelation(method='multipletau', name=(key, k))
      
          for m in ['multipletau', 'tttr2xfcs', 'tttr2xfcs_with_weights']:
              if m in list(ptufile.autoNorm.keys()):
                  for key in list(ptufile.autoNorm[m].keys()):
                      ptufile.save_autocorrelation(name=key, method=m,
                                                   output_path=output_path)
          del ptufile
      
      
      5d0b929a-2855-446b-8b76-55075cd537e3
      
  7. According to log, this run ended with an Error again
    [I 2022-01-23 00:26:40.512 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), keep random ports
    

    Only 229 traces were analyzed.

2.5.10 Analysis 3 of multipletau vs tttr2xfcs, fixed delete_and_shift, weight-random and weight-1-pred

  • log files can be found here:
    tail /beegfs/ye53nis/drmed-git/data/exp-220120-correlate-ptu/2022-02-08_correlate-all-clean-ptu.log
    tail /beegfs/ye53nis/drmed-git/data/exp-220120-correlate-ptu/2022-02-08_correlate-all-dirty-ptu.log
    
    
    
  • This time multiple changes were done in the code. There was an error in correctTCSPC(method='delete_and_shift') which I corrected. I also implemented multipletau fitting with smaller bins than the correction, enabling that we actually get information about these very fast dynamics of the molecules here. Thirdly, to investigate why correction with weights behaved strangely for weights <0.4, I implemented random weighting. Fourthly, to have a correction method which is not dependent on setting a weight manually, I implemented weighting automatically with: weight = 1 - prediction where prediction is the output of the UNET segmentation, a float between 0 and 1.
    1. no correction: I just read in the .ptu data and correlated the photon arrival times
      • 1a using the tttr2xfcs algorithm
      • 1b using the multipletau algorithm
    2. Corrected by photon deletion: I read in the .ptu data, constructed a timetrace from it, fed the time trace to my trained unet which predicts bright burst artifacts, then mapped the time trace prediction on the photon arrival times and deleted all photons inside a bin labelled as artifactual
      • 2a then correlated using the tttr2xfcs algorithm
      • 2b or using the multipletau algorithm
    3. Corrected by photon deletion with shift: I read in the .ptu data, constructed a timetrace from it, fed the timetrace to my trained unet which predicts bright burst artifacts, then mapped the time trace prediction on the photon arrival times and deleted all photons inside a bin labelled as artifactual and shifted the arrival times of all photons which come after a deleted bin by the bin size, then I correlated the resulting new photon arrival times using
      • 3a the tttr2xfcs algorithm
      • 3b the multipletaus algorithm
    4. weight='1-pred': I read in the .ptu data, constructed a timetrace from it, fed the time trace to my trained unet which predicts bright burst artifacts, then mapped the time trace prediction on the photon arrival times and gave all photons inside a bin labelled as artifactual a weight of 1 - prediction in the tttr2xfcs algorithm
    5. weight='random': I read in the .ptu data, constructed a timetrace from it, fed the time trace to my trained unet which predicts bright burst artifacts, then mapped the time trace prediction on the photon arrival times and gave all photons inside a bin labelled as artifactual a random weight between 0 and 1 in the tttr2xfcs algorithm
    6. weight=0 (from 1st exp): I read in the .ptu data, constructed a timetrace from it, fed the time trace to my trained unet which predicts bright burst artifacts, then mapped the time trace prediction on the photon arrival times and gave all photons inside a bin labelled as artifactual a weight of 0.2 in the tttr2xfcs algorithm
    7. weight=0.2 (from 2nd exp): changed weight to 0.2, else see above
    8. weight=0.4 (from 2nd exp): changed weight to 0.4, else see above
    9. weight=0.6 (from 2nd exp): changed weight to 0.6, else see above
    10. weight=0.8 (from 2nd exp): changed weight to 0.8, else see above
  • Then I fitted the traces in each of the 3 folders using Dominic Waithe's https://dwaithe.github.io/FCSfitJS/
    • each correlation was fitted twice with the following options:
      • x: alpha = 1 (fixed) → if we assume no anomalous diffusion
      • y: alpha = 0.5…2 (floating) → values between 0.5…1 fitted the dirty data better
    • the xmin and xmax were:
      • 0.001…100 for clean data (because of shoulder between 100 and 1000 in multipletaudelete)
      • 0.001…1000 for dirty data
    • other parameters:
      • 3D, equation 1B → that means we don't consider txy and tz separately, but have a aspect ratio AR as a conversion between them
      • no of diffusing species: 1
      • triplet states: 1
      • offset: floating
      • GN0: floating
      • A: fixed to 1 (is the relative number of diffusing particles - has to floating with more than 1 species)
      • txy1: floating
      • AR: 5 (values between 4.5 and 6 are okay)
  • now let's look at the results:
    %cd /home/lex/Programme/drmed-git
    
    /home/lex/Programme/drmed-git
    
    import numpy as np
    import matplotlib.pyplot as plt
    import pandas as pd
    import seaborn as sns
    from pathlib import Path
    from pprint import pprint
    # use seaborn style as default even if I just use matplotlib
    sns.set()
    sns.set_palette('colorblind')
    
  • load in all the data (see Details block)
    folder_clean = Path('data/exp-220120-correlate-ptu/2022-02-08_correlate-all-clean-ptu')
    folder_clean_w0 = Path('data/exp-220120-correlate-ptu/2022-01-20_correlate-all-clean-ptu')
    folder_clean_weights = Path('data/exp-220120-correlate-ptu/2022-01-25_correlate-all-clean-ptu')
    folder_dirty = Path('data/exp-220120-correlate-ptu/2022-02-08_correlate-all-dirty-ptu')
    folder_dirty_w0 = Path('data/exp-220120-correlate-ptu/2022-01-20_correlate-all-dirty-ptu')
    folder_dirty_weights = Path('data/exp-220120-correlate-ptu/2022-01-25_correlate-all-dirty-ptu')
    
    # clean params, alphafix
    clean_1ax_param_path = folder_clean / 'clean_tttr2xfcs_no-correction_1comp-alphafix_to100_54of400_param.csv'
    clean_1bx_param_path = folder_clean / 'clean_multipletau_no-correction_1comp-alphafix_to100_54of400_param.csv'
    clean_2ax_param_path = folder_clean / 'clean_tttr2xfcs_delete_1comp-alphafix_to100_54of400_param.csv'
    clean_2bx_param_path = folder_clean / 'clean_multipletau_delete_1comp-alphafix_to100_54of400_param.csv'
    clean_3ax_param_path = folder_clean / 'clean_tttr2xfcs_delete-and-shift_1comp-alphafix_to100_54of400_param.csv'
    clean_3bx_param_path = folder_clean / 'clean_multipletau_delete-and-shift_1comp-alphafix_to100_54of400_param.csv'
    clean_4x_param_path = folder_clean / 'clean_tttr2xfcs_weight-1-pred_1comp-alphafix_to100_54of400_param.csv'
    clean_5x_param_path = folder_clean / 'clean_tttr2xfcs_weight-random_1comp-alphafix_to100_54of400_param.csv'
    
    # clean params, alphafloat
    clean_1ay_param_path = folder_clean / 'clean_tttr2xfcs_no-correction_1comp-alphafloat_to100_54of400_param.csv'
    clean_1by_param_path = folder_clean / 'clean_multipletau_no-correction_1comp-alphafloat_to100_54of400_param.csv'
    clean_2ay_param_path = folder_clean / 'clean_tttr2xfcs_delete_1comp-alphafloat_to100_54of400_param.csv'
    clean_2by_param_path = folder_clean / 'clean_multipletau_delete_1comp-alphafloat_to100_54of400_param.csv'
    clean_3ay_param_path = folder_clean / 'clean_tttr2xfcs_delete-and-shift_1comp-alphafloat_to100_54of400_param.csv'
    clean_3by_param_path = folder_clean / 'clean_multipletau_delete-and-shift_1comp-alphafloat_to100_54of400_param.csv'
    clean_4y_param_path = folder_clean / 'clean_tttr2xfcs_weight-1-pred_1comp-alphafloat_to100_54of400_param.csv'
    clean_5y_param_path = folder_clean / 'clean_tttr2xfcs_weight-random_1comp-alphafloat_to100_54of400_param.csv'
    
    # clean params, from other experiments, alphafix
    clean_6x_param_path = folder_clean_w0 / 'clean_tttr2xfcs_weight-0_1comp-alphafix_to100_229of400_param.csv'
    clean_7x_param_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.2_1comp-alphafix_to100_119of400_param.csv'
    clean_8x_param_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.4_1comp-alphafix_to100_119of400_param.csv'
    clean_9x_param_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.6_1comp-alphafix_to100_119of400_param.csv'
    clean_10x_param_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.8_1comp-alphafix_to100_119of400_param.csv'
    
    # clean params, from other experiments, alphafloat
    clean_6y_param_path = folder_clean_w0 / 'clean_tttr2xfcs_weight-0_1comp-alphafloat_to100_229of400_param.csv'
    clean_7y_param_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.2_1comp-alphafloat_to100_119of400_param.csv'
    clean_8y_param_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.4_1comp-alphafloat_to100_119of400_param.csv'
    clean_9y_param_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.6_1comp-alphafloat_to100_119of400_param.csv'
    clean_10y_param_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.8_1comp-alphafloat_to100_119of400_param.csv'
    
    # clean plots, alphafix
    clean_1ax_plot_path = folder_clean / 'clean_tttr2xfcs_no-correction_1comp-alphafix_to100_54of400_plot.csv'
    clean_1bx_plot_path = folder_clean / 'clean_multipletau_no-correction_1comp-alphafix_to100_54of400_plot.csv'
    clean_2ax_plot_path = folder_clean / 'clean_tttr2xfcs_delete_1comp-alphafix_to100_54of400_plot.csv'
    clean_2bx_plot_path = folder_clean / 'clean_multipletau_delete_1comp-alphafix_to100_54of400_plot.csv'
    clean_3ax_plot_path = folder_clean / 'clean_tttr2xfcs_delete-and-shift_1comp-alphafix_to100_54of400_plot.csv'
    clean_3bx_plot_path = folder_clean / 'clean_multipletau_delete-and-shift_1comp-alphafix_to100_54of400_plot.csv'
    clean_4x_plot_path = folder_clean / 'clean_tttr2xfcs_weight-1-pred_1comp-alphafix_to100_54of400_plot.csv'
    clean_5x_plot_path = folder_clean / 'clean_tttr2xfcs_weight-random_1comp-alphafix_to100_54of400_plot.csv'
    
    # clean plots, alphafloat
    clean_1ay_plot_path = folder_clean / 'clean_tttr2xfcs_no-correction_1comp-alphafloat_to100_54of400_plot.csv'
    clean_1by_plot_path = folder_clean / 'clean_multipletau_no-correction_1comp-alphafloat_to100_54of400_plot.csv'
    clean_2ay_plot_path = folder_clean / 'clean_tttr2xfcs_delete_1comp-alphafloat_to100_54of400_plot.csv'
    clean_2by_plot_path = folder_clean / 'clean_multipletau_delete_1comp-alphafloat_to100_54of400_plot.csv'
    clean_3ay_plot_path = folder_clean / 'clean_tttr2xfcs_delete-and-shift_1comp-alphafloat_to100_54of400_plot.csv'
    clean_3by_plot_path = folder_clean / 'clean_multipletau_delete-and-shift_1comp-alphafloat_to100_54of400_plot.csv'
    clean_4y_plot_path = folder_clean / 'clean_tttr2xfcs_weight-1-pred_1comp-alphafloat_to100_54of400_plot.csv'
    clean_5y_plot_path = folder_clean / 'clean_tttr2xfcs_weight-random_1comp-alphafloat_to100_54of400_plot.csv'
    
    # clean plots, from other experiments, alphafix
    clean_6x_plot_path = folder_clean_w0 / 'clean_tttr2xfcs_weight-0_1comp-alphafix_to100_229of400_plot.csv'
    clean_7x_plot_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.2_1comp-alphafix_to100_119of400_plot.csv'
    clean_8x_plot_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.4_1comp-alphafix_to100_119of400_plot.csv'
    clean_9x_plot_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.6_1comp-alphafix_to100_119of400_plot.csv'
    clean_10x_plot_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.8_1comp-alphafix_to100_119of400_plot.csv'
    
    # clean plots, from other experiments, alphafix
    clean_6y_plot_path = folder_clean_w0 / 'clean_tttr2xfcs_weight-0_1comp-alphafloat_to100_229of400_plot.csv'
    clean_7y_plot_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.2_1comp-alphafloat_to100_119of400_plot.csv'
    clean_8y_plot_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.4_1comp-alphafloat_to100_119of400_plot.csv'
    clean_9y_plot_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.6_1comp-alphafloat_to100_119of400_plot.csv'
    clean_10y_plot_path = folder_clean_weights / 'clean_tttr2xfcs_weight-0.8_1comp-alphafloat_to100_119of400_plot.csv'
    
    # dirty params, alphafix
    dirty_1ax_param_path = folder_dirty / 'dirty_tttr2xfcs_no-correction_1comp-alphafix_to1000_53of400_param.csv'
    dirty_1bx_param_path = folder_dirty / 'dirty_multipletau_no-correction_1comp-alphafix_to1000_53of400_param.csv'
    dirty_2ax_param_path = folder_dirty / 'dirty_tttr2xfcs_delete_1comp-alphafix_to1000_53of400_param.csv'
    dirty_2bx_param_path = folder_dirty / 'dirty_multipletau_delete_1comp-alphafix_to1000_53of400_param.csv'
    dirty_3ax_param_path = folder_dirty / 'dirty_tttr2xfcs_delete-and-shift_1comp-alphafix_to1000_53of400_param.csv'
    dirty_3bx_param_path = folder_dirty / 'dirty_multipletau_delete-and-shift_1comp-alphafix_to1000_53of400_param.csv'
    dirty_4x_param_path = folder_dirty / 'dirty_tttr2xfcs_weight-1-pred_1comp-alphafix_to1000_53of400_param.csv'
    dirty_5x_param_path = folder_dirty / 'dirty_tttr2xfcs_weight-random_1comp-alphafix_to1000_53of400_param.csv'
    
    # dirty params, alphafloat
    dirty_1ay_param_path = folder_dirty / 'dirty_tttr2xfcs_no-correction_1comp-alphafloat_to1000_53of400_param.csv'
    dirty_1by_param_path = folder_dirty / 'dirty_multipletau_no-correction_1comp-alphafloat_to1000_53of400_param.csv'
    dirty_2ay_param_path = folder_dirty / 'dirty_tttr2xfcs_delete_1comp-alphafloat_to1000_53of400_param.csv'
    dirty_2by_param_path = folder_dirty / 'dirty_multipletau_delete_1comp-alphafloat_to1000_53of400_param.csv'
    dirty_3ay_param_path = folder_dirty / 'dirty_tttr2xfcs_delete-and-shift_1comp-alphafloat_to1000_53of400_param.csv'
    dirty_3by_param_path = folder_dirty / 'dirty_multipletau_delete-and-shift_1comp-alphafloat_to1000_53of400_param.csv'
    dirty_4y_param_path = folder_dirty / 'dirty_tttr2xfcs_weight-1-pred_1comp-alphafloat_to1000_53of400_param.csv'
    dirty_5y_param_path = folder_dirty / 'dirty_tttr2xfcs_weight-random_1comp-alphafloat_to1000_53of400_param.csv'
    
    # dirty params, from other experiments, alphafix
    dirty_6x_param_path = folder_dirty_w0 / 'dirty_tttr2xfcs_weight-0_1comp-alphafix_to1000_173of400_param.csv'
    dirty_7x_param_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.2_1comp-alphafix_to1000_79of400_param.csv'
    dirty_8x_param_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.4_1comp-alphafix_to1000_79of400_param.csv'
    dirty_9x_param_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.6_1comp-alphafix_to1000_79of400_param.csv'
    dirty_10x_param_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.8_1comp-alphafix_to1000_79of400_param.csv'
    
    # dirty params, from other experiments, alphafloat
    dirty_6y_param_path = folder_dirty_w0 / 'dirty_tttr2xfcs_weight-0_1comp-alphafloat_to1000_173of400_param.csv'
    dirty_7y_param_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.2_1comp-alphafloat_to1000_79of400_param.csv'
    dirty_8y_param_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.4_1comp-alphafloat_to1000_79of400_param.csv'
    dirty_9y_param_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.6_1comp-alphafloat_to1000_79of400_param.csv'
    dirty_10y_param_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.8_1comp-alphafloat_to1000_79of400_param.csv'
    
    # dirty plots, alphafix
    dirty_1ax_plot_path = folder_dirty / 'dirty_tttr2xfcs_no-correction_1comp-alphafix_to1000_53of400_plot.csv'
    dirty_1bx_plot_path = folder_dirty / 'dirty_multipletau_no-correction_1comp-alphafix_to1000_53of400_plot.csv'
    dirty_2ax_plot_path = folder_dirty / 'dirty_tttr2xfcs_delete_1comp-alphafix_to1000_53of400_plot.csv'
    dirty_2bx_plot_path = folder_dirty / 'dirty_multipletau_delete_1comp-alphafix_to1000_53of400_plot.csv'
    dirty_3ax_plot_path = folder_dirty / 'dirty_tttr2xfcs_delete-and-shift_1comp-alphafix_to1000_53of400_plot.csv'
    dirty_3bx_plot_path = folder_dirty / 'dirty_multipletau_delete-and-shift_1comp-alphafix_to1000_53of400_plot.csv'
    dirty_4x_plot_path = folder_dirty / 'dirty_tttr2xfcs_weight-1-pred_1comp-alphafix_to1000_53of400_plot.csv'
    dirty_5x_plot_path = folder_dirty / 'dirty_tttr2xfcs_weight-random_1comp-alphafix_to1000_53of400_plot.csv'
    
    # dirty plots, alphafloat
    dirty_1ay_plot_path = folder_dirty / 'dirty_tttr2xfcs_no-correction_1comp-alphafloat_to1000_53of400_plot.csv'
    dirty_1by_plot_path = folder_dirty / 'dirty_multipletau_no-correction_1comp-alphafloat_to1000_53of400_plot.csv'
    dirty_2ay_plot_path = folder_dirty / 'dirty_tttr2xfcs_delete_1comp-alphafloat_to1000_53of400_plot.csv'
    dirty_2by_plot_path = folder_dirty / 'dirty_multipletau_delete_1comp-alphafloat_to1000_53of400_plot.csv'
    dirty_3ay_plot_path = folder_dirty / 'dirty_tttr2xfcs_delete-and-shift_1comp-alphafloat_to1000_53of400_plot.csv'
    dirty_3by_plot_path = folder_dirty / 'dirty_multipletau_delete-and-shift_1comp-alphafloat_to1000_53of400_plot.csv'
    dirty_4y_plot_path = folder_dirty / 'dirty_tttr2xfcs_weight-1-pred_1comp-alphafloat_to1000_53of400_plot.csv'
    dirty_5y_plot_path = folder_dirty / 'dirty_tttr2xfcs_weight-random_1comp-alphafloat_to1000_53of400_plot.csv'
    
    # dirty plot, from other experiments, alphafix
    dirty_6x_plot_path = folder_dirty_w0 / 'dirty_tttr2xfcs_weight-0_1comp-alphafix_to1000_173of400_plot.csv'
    dirty_7x_plot_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.2_1comp-alphafix_to1000_79of400_plot.csv'
    dirty_8x_plot_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.4_1comp-alphafix_to1000_79of400_plot.csv'
    dirty_9x_plot_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.6_1comp-alphafix_to1000_79of400_plot.csv'
    dirty_10x_plot_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.8_1comp-alphafix_to1000_79of400_plot.csv'
    
    # dirty plot, from other experiments, alphafloat
    dirty_6y_plot_path = folder_dirty_w0 / 'dirty_tttr2xfcs_weight-0_1comp-alphafloat_to1000_173of400_plot.csv'
    dirty_7y_plot_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.2_1comp-alphafloat_to1000_79of400_plot.csv'
    dirty_8y_plot_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.4_1comp-alphafloat_to1000_79of400_plot.csv'
    dirty_9y_plot_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.6_1comp-alphafloat_to1000_79of400_plot.csv'
    dirty_10y_plot_path = folder_dirty_weights / 'dirty_tttr2xfcs_weight-0.8_1comp-alphafloat_to1000_79of400_plot.csv'
    
    
    # clean params, alphafix
    clean_1ax_param = pd.read_csv(clean_1ax_param_path, sep=',').assign(
        fit_with=54*['alpha=1',], correlator=54*['tttr2xfcs',],
        artifact=54*['clean',], correction=54*['no correction',])
    clean_1bx_param = pd.read_csv(clean_1bx_param_path, sep=',').assign(
        fit_with=54*['alpha=1',], correlator=54*['multipletau',],
        artifact=54*['clean',], correction=54*['no correction',])
    clean_2ax_param = pd.read_csv(clean_2ax_param_path, sep=',').assign(
        fit_with=54*['alpha=1',], correlator=54*['tttr2xfcs',],
        artifact=54*['clean',], correction=54*['delete',])
    clean_2bx_param = pd.read_csv(clean_2bx_param_path, sep=',').assign(
        fit_with=54*['alpha=1',], correlator=54*['multipletau',],
        artifact=54*['clean',], correction=54*['delete',])
    clean_3ax_param = pd.read_csv(clean_3ax_param_path, sep=',').assign(
        fit_with=54*['alpha=1',], correlator=54*['tttr2xfcs',],
        artifact=54*['clean',], correction=54*['delete and shift',])
    clean_3bx_param = pd.read_csv(clean_3bx_param_path, sep=',').assign(
        fit_with=54*['alpha=1',], correlator=54*['multipletau',],
        artifact=54*['clean',], correction=54*['delete and shift',])
    clean_4x_param =  pd.read_csv(clean_4x_param_path, sep=',').assign(
        fit_with=54*['alpha=1',], correlator=54*['tttr2xfcs',],
        artifact=54*['clean',], correction=54*['weight=1-pred',])
    clean_5x_param =  pd.read_csv(clean_5x_param_path, sep=',').assign(
        fit_with=54*['alpha=1',], correlator=54*['tttr2xfcs',],
        artifact=54*['clean',], correction=54*['weight=random',])
    
    # clean params, alphafloat
    clean_1ay_param = pd.read_csv(clean_1ay_param_path, sep=',').assign(
        fit_with=54*['alpha=float',], correlator=54*['tttr2xfcs',],
        artifact=54*['clean',], correction=54*['no correction',])
    clean_1by_param = pd.read_csv(clean_1by_param_path, sep=',').assign(
        fit_with=54*['alpha=float',], correlator=54*['multipletau',],
        artifact=54*['clean',], correction=54*['no correction',])
    clean_2ay_param = pd.read_csv(clean_2ay_param_path, sep=',').assign(
        fit_with=54*['alpha=float',], correlator=54*['tttr2xfcs',],
        artifact=54*['clean',], correction=54*['delete',])
    clean_2by_param = pd.read_csv(clean_2by_param_path, sep=',').assign(
        fit_with=54*['alpha=float',], correlator=54*['multipletau',],
        artifact=54*['clean',], correction=54*['delete',])
    clean_3ay_param = pd.read_csv(clean_3ay_param_path, sep=',').assign(
        fit_with=54*['alpha=float',], correlator=54*['tttr2xfcs',],
        artifact=54*['clean',], correction=54*['delete and shift',])
    clean_3by_param = pd.read_csv(clean_3by_param_path, sep=',').assign(
        fit_with=54*['alpha=float',], correlator=54*['multipletau',],
        artifact=54*['clean',], correction=54*['delete and shift',])
    clean_4y_param =  pd.read_csv(clean_4y_param_path, sep=',').assign(
        fit_with=54*['alpha=float',], correlator=54*['tttr2xfcs',],
        artifact=54*['clean',], correction=54*['weight=1-pred',])
    clean_5y_param =  pd.read_csv(clean_5y_param_path, sep=',').assign(
        fit_with=54*['alpha=float',], correlator=54*['tttr2xfcs',],
        artifact=54*['clean',], correction=54*['weight=random',])
    
    # clean params, from other experiments, alphafix
    clean_6x_param =  pd.read_csv(clean_6x_param_path, sep=',').assign(
        fit_with=229*['alpha=1',], correlator=229*['tttr2xfcs',],
        artifact=229*['clean',], correction=229*['weight=0',])
    clean_7x_param =  pd.read_csv(clean_7x_param_path, sep=',').assign(
        fit_with=119*['alpha=1',], correlator=119*['tttr2xfcs',],
        artifact=119*['clean',], correction=119*['weight=0.2',])
    clean_8x_param =  pd.read_csv(clean_8x_param_path, sep=',').assign(
        fit_with=119*['alpha=1',], correlator=119*['tttr2xfcs',],
        artifact=119*['clean',], correction=119*['weight=0.4',])
    clean_9x_param =  pd.read_csv(clean_9x_param_path, sep=',').assign(
        fit_with=119*['alpha=1',], correlator=119*['tttr2xfcs',],
        artifact=119*['clean',], correction=119*['weight=0.6',])
    clean_10x_param = pd.read_csv(clean_10x_param_path, sep=',').assign(
        fit_with=119*['alpha=1',], correlator=119*['tttr2xfcs',],
        artifact=119*['clean',], correction=119*['weight=0.8',])
    
    # clean params, from other experiments, alphafloat
    clean_6y_param =  pd.read_csv(clean_6y_param_path, sep=',').assign(
        fit_with=229*['alpha=float',], correlator=229*['tttr2xfcs',],
        artifact=229*['clean',], correction=229*['weight=0',])
    clean_7y_param =  pd.read_csv(clean_7y_param_path, sep=',').assign(
        fit_with=119*['alpha=float',], correlator=119*['tttr2xfcs',],
        artifact=119*['clean',], correction=119*['weight=0.2',])
    clean_8y_param =  pd.read_csv(clean_8y_param_path, sep=',').assign(
        fit_with=119*['alpha=float',], correlator=119*['tttr2xfcs',],
        artifact=119*['clean',], correction=119*['weight=0.4',])
    clean_9y_param =  pd.read_csv(clean_9y_param_path, sep=',').assign(
        fit_with=119*['alpha=float',], correlator=119*['tttr2xfcs',],
        artifact=119*['clean',], correction=119*['weight=0.6',])
    clean_10y_param = pd.read_csv(clean_10y_param_path, sep=',').assign(
        fit_with=119*['alpha=float',], correlator=119*['tttr2xfcs',],
        artifact=119*['clean',], correction=119*['weight=0.8',])
    
    # dirty params, alphafix
    dirty_1ax_param = pd.read_csv(dirty_1ax_param_path, sep=',').assign(
        fit_with=53*['alpha=1',], correlator=53*['tttr2xfcs',],
        artifact=53*['dirty',], correction=53*['no correction',])
    dirty_1bx_param = pd.read_csv(dirty_1bx_param_path, sep=',').assign(
        fit_with=53*['alpha=1',], correlator=53*['multipletau',],
        artifact=53*['dirty',], correction=53*['no correction',])
    dirty_2ax_param = pd.read_csv(dirty_2ax_param_path, sep=',').assign(
        fit_with=53*['alpha=1',], correlator=53*['tttr2xfcs',],
        artifact=53*['dirty',], correction=53*['delete',])
    dirty_2bx_param = pd.read_csv(dirty_2bx_param_path, sep=',').assign(
        fit_with=53*['alpha=1',], correlator=53*['multipletau',],
        artifact=53*['dirty',], correction=53*['delete',])
    dirty_3ax_param = pd.read_csv(dirty_3ax_param_path, sep=',').assign(
        fit_with=53*['alpha=1',], correlator=53*['tttr2xfcs',],
        artifact=53*['dirty',], correction=53*['delete and shift',])
    dirty_3bx_param = pd.read_csv(dirty_3bx_param_path, sep=',').assign(
        fit_with=53*['alpha=1',], correlator=53*['multipletau',],
        artifact=53*['dirty',], correction=53*['delete and shift',])
    dirty_4x_param =  pd.read_csv(dirty_4x_param_path, sep=',').assign(
        fit_with=53*['alpha=1',], correlator=53*['tttr2xfcs',],
        artifact=53*['dirty',], correction=53*['weight=1-pred',])
    dirty_5x_param =  pd.read_csv(dirty_5x_param_path, sep=',').assign(
        fit_with=53*['alpha=1',], correlator=53*['tttr2xfcs',],
        artifact=53*['dirty',], correction=53*['weight=random',])
    
    # dirty params, alphafloat
    dirty_1ay_param = pd.read_csv(dirty_1ay_param_path, sep=',').assign(
        fit_with=53*['alpha=float',], correlator=53*['tttr2xfcs',],
        artifact=53*['dirty',], correction=53*['no correction',])
    dirty_1by_param = pd.read_csv(dirty_1by_param_path, sep=',').assign(
        fit_with=53*['alpha=float',], correlator=53*['multipletau',],
        artifact=53*['dirty',], correction=53*['no correction',])
    dirty_2ay_param = pd.read_csv(dirty_2ay_param_path, sep=',').assign(
        fit_with=53*['alpha=float',], correlator=53*['tttr2xfcs',],
        artifact=53*['dirty',], correction=53*['delete',])
    dirty_2by_param = pd.read_csv(dirty_2by_param_path, sep=',').assign(
        fit_with=53*['alpha=float',], correlator=53*['multipletau',],
        artifact=53*['dirty',], correction=53*['delete',])
    dirty_3ay_param = pd.read_csv(dirty_3ay_param_path, sep=',').assign(
        fit_with=53*['alpha=float',], correlator=53*['tttr2xfcs',],
        artifact=53*['dirty',], correction=53*['delete and shift',])
    dirty_3by_param = pd.read_csv(dirty_3by_param_path, sep=',').assign(
        fit_with=53*['alpha=float',], correlator=53*['multipletau',],
        artifact=53*['dirty',], correction=53*['delete and shift',])
    dirty_4y_param =  pd.read_csv(dirty_4y_param_path, sep=',').assign(
        fit_with=53*['alpha=float',], correlator=53*['tttr2xfcs',],
        artifact=53*['dirty',], correction=53*['weight=1-pred',])
    dirty_5y_param =  pd.read_csv(dirty_5y_param_path, sep=',').assign(
        fit_with=53*['alpha=float',], correlator=53*['tttr2xfcs',],
        artifact=53*['dirty',], correction=53*['weight=random',])
    
    # dirty params, from other experiments, alphafix
    dirty_6x_param =  pd.read_csv(dirty_6x_param_path, sep=',').assign(
        fit_with=173*['alpha=1',], correlator=173*['tttr2xfcs',],
        artifact=173*['dirty',], correction=173*['weight=0',])
    dirty_7x_param =  pd.read_csv(dirty_7x_param_path, sep=',').assign(
        fit_with=79*['alpha=1',], correlator=79*['tttr2xfcs',],
        artifact=79*['dirty',], correction=79*['weight=0.2',])
    dirty_8x_param =  pd.read_csv(dirty_8x_param_path, sep=',').assign(
        fit_with=79*['alpha=1',], correlator=79*['tttr2xfcs',],
        artifact=79*['dirty',], correction=79*['weight=0.4',])
    dirty_9x_param =  pd.read_csv(dirty_9x_param_path, sep=',').assign(
        fit_with=79*['alpha=1',], correlator=79*['tttr2xfcs',],
        artifact=79*['dirty',], correction=79*['weight=0.6',])
    dirty_10x_param = pd.read_csv(dirty_10x_param_path, sep=',').assign(
        fit_with=79*['alpha=1',], correlator=79*['tttr2xfcs',],
        artifact=79*['dirty',], correction=79*['weight=0.8',])
    
    # dirty params, from other experiments, alphafloat
    dirty_6y_param =  pd.read_csv(dirty_6y_param_path, sep=',').assign(
        fit_with=173*['alpha=float',], correlator=173*['tttr2xfcs',],
        artifact=173*['dirty',], correction=173*['weight=0',])
    dirty_7y_param =  pd.read_csv(dirty_7y_param_path, sep=',').assign(
        fit_with=79*['alpha=float',], correlator=79*['tttr2xfcs',],
        artifact=79*['dirty',], correction=79*['weight=0.2',])
    dirty_8y_param =  pd.read_csv(dirty_8y_param_path, sep=',').assign(
        fit_with=79*['alpha=float',], correlator=79*['tttr2xfcs',],
        artifact=79*['dirty',], correction=79*['weight=0.4',])
    dirty_9y_param =  pd.read_csv(dirty_9y_param_path, sep=',').assign(
        fit_with=79*['alpha=float',], correlator=79*['tttr2xfcs',],
        artifact=79*['dirty',], correction=79*['weight=0.6',])
    dirty_10y_param = pd.read_csv(dirty_10y_param_path, sep=',').assign(
        fit_with=79*['alpha=float',], correlator=79*['tttr2xfcs',],
        artifact=79*['dirty',], correction=79*['weight=0.8',])
    
    all_param = pd.concat([clean_1ax_param, clean_1bx_param, clean_2ax_param,
                           clean_2bx_param, clean_3ax_param, clean_3bx_param,
                           clean_4x_param, clean_5x_param, clean_6x_param,
                           clean_7x_param, clean_8x_param, clean_9x_param,
                           clean_10x_param, clean_1ay_param, clean_1by_param,
                           clean_2ay_param, clean_2by_param, clean_3ay_param,
                           clean_3by_param, clean_4y_param, clean_5y_param,
                           clean_6y_param, clean_7y_param, clean_8y_param,
                           clean_9y_param, clean_10y_param, dirty_1ax_param,
                           dirty_1bx_param, dirty_2ax_param, dirty_2bx_param,
                           dirty_3ax_param, dirty_3bx_param, dirty_4x_param,
                           dirty_5x_param, dirty_6x_param, dirty_7x_param,
                           dirty_8x_param, dirty_9x_param, dirty_10x_param,
                           dirty_1ay_param, dirty_1by_param, dirty_2ay_param,
                           dirty_2by_param, dirty_3ay_param, dirty_3by_param,
                           dirty_4y_param, dirty_5y_param, dirty_6y_param,
                           dirty_7y_param, dirty_8y_param, dirty_9y_param,
                           dirty_10y_param], ignore_index=True)
    
    all_param["correlator-fit_with"] = all_param[["correlator", "fit_with"]].agg(' - '.join, axis=1)
    all_param
    
      nameofplot masterfile parentname parentuqid time of fit Diffeq Diffspecies Tripleteq Tripletspecies Dimen stdev(AR1) alpha1 stdev(alpha1) N (mom) bri (kHz) fitwith correlator artifact correction correlator-fitwith
    0 2022-02-10tttr2xfcsCH2BIN1dot020 nM AF4881… Not known tttr2xfcs NaN 11 February 2022 14:02:13 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 1.000000 NaN 72.053415 4.794626 alpha=1 tttr2xfcs clean no correction tttr2xfcs - alpha=1
    1 2022-02-10tttr2xfcsCH2BIN1dot020 nM AF4882… Not known tttr2xfcs NaN 11 February 2022 14:02:13 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 1.000000 NaN 73.936945 4.666217 alpha=1 tttr2xfcs clean no correction tttr2xfcs - alpha=1
    2 2022-02-10tttr2xfcsCH2BIN1dot020 nM AF4882… Not known tttr2xfcs NaN 11 February 2022 14:02:13 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 1.000000 NaN 70.721472 4.901863 alpha=1 tttr2xfcs clean no correction tttr2xfcs - alpha=1
    3 2022-02-10tttr2xfcsCH2BIN1dot020 nM AF4882… Not known tttr2xfcs NaN 11 February 2022 14:02:13 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 1.000000 NaN 72.151740 4.797506 alpha=1 tttr2xfcs clean no correction tttr2xfcs - alpha=1
    4 2022-02-10tttr2xfcsCH2BIN1dot020 nM AF4883… Not known tttr2xfcs NaN 11 February 2022 14:02:13 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 1.000000 NaN 72.730578 4.762291 alpha=1 tttr2xfcs clean no correction tttr2xfcs - alpha=1
    4095 2022-01-26tttr2xfcswithweightsCH2BIN1wei Not known weight0.8DiO LUV 10uM in 20 nM AF488247T2974s1 NaN 13 February 2022 18:58:28 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 0.500000 NaN 8.066484 61.800612 alpha=float tttr2xfcs dirty weight=0.8 tttr2xfcs - alpha=float
    4096 2022-01-26tttr2xfcswithweightsCH2BIN1wei Not known weight0.8DiO LUV 10uM in 20 nM AF488248T2986s1 NaN 13 February 2022 18:58:28 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 0.594970 NaN 4.674732 113.075760 alpha=float tttr2xfcs dirty weight=0.8 tttr2xfcs - alpha=float
    4097 2022-01-26tttr2xfcswithweightsCH2BIN1wei Not known weight0.8DiO LUV 10uM in 20 nM AF488291T3505s1 NaN 13 February 2022 18:58:28 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 0.791023 NaN 3.326365 172.746385 alpha=float tttr2xfcs dirty weight=0.8 tttr2xfcs - alpha=float
    4098 2022-01-26tttr2xfcswithweightsCH2BIN1wei Not known weight0.8DiO LUV 10uM in 20 nM AF488293T3529s1 NaN 13 February 2022 18:58:28 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 0.500000 NaN 9.386688 53.060101 alpha=float tttr2xfcs dirty weight=0.8 tttr2xfcs - alpha=float
    4099 2022-01-26tttr2xfcswithweightsCH2BIN1wei Not known weight0.8DiO LUV 10uM in 20 nM AF488294T3541s1 NaN 13 February 2022 18:58:28 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 0.500000 NaN 6.441874 83.167622 alpha=float tttr2xfcs dirty weight=0.8 tttr2xfcs - alpha=float

    4100 rows × 33 columns

  • let's plot all transit times for each correction method and parameters
    g = sns.FacetGrid(data=all_param,
                      row='artifact',
                      sharey=True,
                      sharex=True,
                      height=5,
                      aspect=2,
                      legend_out=True)
    g.map_dataframe(sns.boxplot,
          y='txy1',
          x='correction',
          hue="correlator-fit_with",
          palette='colorblind',
          showfliers=False)
    g.add_legend(title='Correlator and\nfit parameter')
    g.map_dataframe(sns.stripplot,
          y='txy1',
          x='correction',
          hue="correlator-fit_with",
          dodge=True,
          palette=sns.color_palette(['0.3']),
          size=4,
          jitter=0.2)
    g.tight_layout()
    g.fig.suptitle('AlexaFluor488 (top) vs AlexaFluor488 + Dio LUVs (bottom)',
                    y=1.03, size=20)
    for axes in g.axes.flat:
         _ = axes.set_xticklabels(axes.get_xticklabels(), rotation=45)
    plt.setp(g.axes, yscale='log', xlabel='correction method',
             ylabel=r'transit time $\tau_{D}$ (log)')
    plt.show()
    
  • we'll use the more concise seaborn.catplot for plotting:
    g = sns.catplot(data=all_param,
                    y='txy1',
                    x='correction',
                    hue='correlator-fit_with',
                    row='artifact',
                    sharey=True,
                    height=5,
                    aspect=2,
                    legend_out=True,
                    kind='boxen',
                    showfliers=False)
    g.map_dataframe(sns.stripplot,
          y='txy1',
          x='correction',
          hue="correlator-fit_with",
          dodge=True,
          palette=sns.color_palette(['0.3']),
          size=4,
          jitter=0.2)
    g.tight_layout()
    g.fig.suptitle('AlexaFluor488 (top) vs AlexaFluor488 + Dio LUVs (bottom)',
                   y=1.03, size=20)
    for axes in g.axes.flat:
         _ = axes.set_xticklabels(axes.get_xticklabels(), rotation=45)
    plt.setp(g.axes, yscale='log', xlabel='correction method',
             ylabel=r'transit time $\tau_{D}$ (log)')
    plt.setp(g.legend, title='Correlator and\nfit parameter')
    plt.show()
    

    analysis3-all-param.png

  • for sake of simplicity, add a plot of only no correction vs delete and shift:
    data = all_param.loc[all_param['correction'].isin(['no correction', 'delete and shift'])]
    display(data)
    g = sns.catplot(data=data,
                    y='txy1',
                    x='correction',
                    hue='correlator-fit_with',
                    col='artifact',
                    sharey=True,
                    height=5,
                    aspect=1,
                    legend_out=True,
                    kind='boxen',
                    showfliers=False)
    g.map_dataframe(sns.stripplot,
          y='txy1',
          x='correction',
          hue="correlator-fit_with",
          dodge=True,
          palette=sns.color_palette(['0.3']),
          size=4,
          jitter=0.2)
    g.tight_layout()
    g.fig.suptitle('AlexaFluor488 (left) vs AlexaFluor488 + Dio LUVs (right)', y=1.03, size=20)
    for axes in g.axes.flat:
         _ = axes.set_xticklabels(axes.get_xticklabels(), rotation=45)
    plt.setp(g.axes, yscale='log', xlabel='correction method',
             ylabel=r'transit time $\tau_{D}$ (log)')
    plt.setp(g.legend, title='Correlator and\nfit parameter')
    plt.show()
    
    
      nameofplot masterfile parentname parentuqid time of fit Diffeq Diffspecies Tripleteq Tripletspecies Dimen stdev(AR1) alpha1 stdev(alpha1) N (mom) bri (kHz) fitwith correlator artifact correction correlator-fitwith
    0 2022-02-10tttr2xfcsCH2BIN1dot020 nM AF4881… Not known tttr2xfcs NaN 11 February 2022 14:02:13 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 1.000000 NaN 72.053415 4.794626 alpha=1 tttr2xfcs clean no correction tttr2xfcs - alpha=1
    1 2022-02-10tttr2xfcsCH2BIN1dot020 nM AF4882… Not known tttr2xfcs NaN 11 February 2022 14:02:13 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 1.000000 NaN 73.936945 4.666217 alpha=1 tttr2xfcs clean no correction tttr2xfcs - alpha=1
    2 2022-02-10tttr2xfcsCH2BIN1dot020 nM AF4882… Not known tttr2xfcs NaN 11 February 2022 14:02:13 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 1.000000 NaN 70.721472 4.901863 alpha=1 tttr2xfcs clean no correction tttr2xfcs - alpha=1
    3 2022-02-10tttr2xfcsCH2BIN1dot020 nM AF4882… Not known tttr2xfcs NaN 11 February 2022 14:02:13 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 1.000000 NaN 72.151740 4.797506 alpha=1 tttr2xfcs clean no correction tttr2xfcs - alpha=1
    4 2022-02-10tttr2xfcsCH2BIN1dot020 nM AF4883… Not known tttr2xfcs NaN 11 February 2022 14:02:13 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 1.000000 NaN 72.730578 4.762291 alpha=1 tttr2xfcs clean no correction tttr2xfcs - alpha=1
    3500 2022-02-10multipletauCH2BIN0dot001usDiO L… Not known multipletau NaN 12 February 2022 21:01:19 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 0.500000 NaN -9.807292 -45.831485 alpha=float multipletau dirty delete and shift multipletau - alpha=float
    3501 2022-02-10multipletauCH2BIN0dot001usDiO L… Not known multipletau NaN 12 February 2022 21:01:19 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 0.500000 NaN -9.508777 -50.024738 alpha=float multipletau dirty delete and shift multipletau - alpha=float
    3502 2022-02-10multipletauCH2BIN0dot001usDiO L… Not known multipletau NaN 12 February 2022 21:01:19 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 0.647205 NaN -9.413903 -47.210210 alpha=float multipletau dirty delete and shift multipletau - alpha=float
    3503 2022-02-10multipletauCH2BIN0dot001usDiO L… Not known multipletau NaN 12 February 2022 21:01:19 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 0.500000 NaN -9.666384 -50.652535 alpha=float multipletau dirty delete and shift multipletau - alpha=float
    3504 2022-02-10multipletauCH2BIN0dot001usDiO L… Not known multipletau NaN 12 February 2022 21:01:19 Equation 1B 1 Triplet Eq 2A 1 NaN NaN 0.500000 NaN -9.805858 -48.478584 alpha=float multipletau dirty delete and shift multipletau - alpha=float

    856 rows × 33 columns

    analysis3-nocorr-vs-delete-and-shift.png

  • let's save the DataFrame used for plotting so that re-using it is simpler:
    all_param.to_csv('data/exp-220120-correlate-ptu/2022-02-13_all-params.csv')
    
  • We saw above that the correction methods lead to quite different results regarding the transit time outcomes. The computation of the transit times depends on:
    1. the artifact prediction on the time trace (and subsequent correction of photons)
    2. the correlation of the photon arrival times (or the binned timetrace)
    3. the fitting of the correlation
  • let's have a deeper look in the correlations and fits, first load all the correlation and fit plots which were done in https://dwaithe.github.io/FCSfitJS/
    # clean plots, alphafix
    clean_1ax_plot = pd.read_csv(clean_1ax_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    clean_1bx_plot = pd.read_csv(clean_1bx_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    clean_2ax_plot = pd.read_csv(clean_2ax_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    clean_2bx_plot = pd.read_csv(clean_2bx_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    clean_3ax_plot = pd.read_csv(clean_3ax_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    clean_3bx_plot = pd.read_csv(clean_3bx_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    clean_4x_plot =  pd.read_csv(clean_4x_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    clean_5x_plot =  pd.read_csv(clean_5x_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    
    # clean plots, alphafloat
    clean_1ay_plot = pd.read_csv(clean_1ay_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    clean_1by_plot = pd.read_csv(clean_1by_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    clean_2ay_plot = pd.read_csv(clean_2ay_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    clean_2by_plot = pd.read_csv(clean_2by_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    clean_3ay_plot = pd.read_csv(clean_3ay_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    clean_3by_plot = pd.read_csv(clean_3by_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    clean_4y_plot =  pd.read_csv(clean_4y_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    clean_5y_plot =  pd.read_csv(clean_5y_plot_path, sep=',', na_values=' ').drop('Unnamed: 109', axis=1)
    
    # clean plots, from other experiments, alphafix
    clean_6x_plot =  pd.read_csv(clean_6x_plot_path, sep=',', na_values=' ').drop('Unnamed: 459', axis=1)
    clean_7x_plot =  pd.read_csv(clean_7x_plot_path, sep=',', na_values=' ').drop('Unnamed: 239', axis=1)
    clean_8x_plot =  pd.read_csv(clean_8x_plot_path, sep=',', na_values=' ').drop('Unnamed: 239', axis=1)
    clean_9x_plot =  pd.read_csv(clean_9x_plot_path, sep=',', na_values=' ').drop('Unnamed: 239', axis=1)
    clean_10x_plot = pd.read_csv(clean_10x_plot_path, sep=',', na_values=' ').drop('Unnamed: 239', axis=1)
    
    # clean plots, from other experiments, alphafloat
    clean_6y_plot =  pd.read_csv(clean_6y_plot_path, sep=',', na_values=' ').drop('Unnamed: 459', axis=1)
    clean_7y_plot =  pd.read_csv(clean_7y_plot_path, sep=',', na_values=' ').drop('Unnamed: 239', axis=1)
    clean_8y_plot =  pd.read_csv(clean_8y_plot_path, sep=',', na_values=' ').drop('Unnamed: 239', axis=1)
    clean_9y_plot =  pd.read_csv(clean_9y_plot_path, sep=',', na_values=' ').drop('Unnamed: 239', axis=1)
    clean_10y_plot = pd.read_csv(clean_10y_plot_path, sep=',', na_values=' ').drop('Unnamed: 239', axis=1)
    
    # clean transit times, alphafix
    clean_1ax_tau = clean_1ax_plot['Time (ms)']
    clean_1bx_tau = clean_1bx_plot['Time (ms)']
    clean_2ax_tau = clean_2ax_plot['Time (ms)']
    clean_2bx_tau = clean_2bx_plot['Time (ms)']
    clean_3ax_tau = clean_3ax_plot['Time (ms)']
    clean_3bx_tau = clean_3bx_plot['Time (ms)']
    clean_4x_tau =  clean_4x_plot['Time (ms)']
    clean_5x_tau =  clean_5x_plot['Time (ms)']
    
    # clean transit times, alphafloat
    clean_1ay_tau = clean_1ay_plot['Time (ms)']
    clean_1by_tau = clean_1by_plot['Time (ms)']
    clean_2ay_tau = clean_2ay_plot['Time (ms)']
    clean_2by_tau = clean_2by_plot['Time (ms)']
    clean_3ay_tau = clean_3ay_plot['Time (ms)']
    clean_3by_tau = clean_3by_plot['Time (ms)']
    clean_4y_tau =  clean_4y_plot['Time (ms)']
    clean_5y_tau =  clean_5y_plot['Time (ms)']
    
    # clean transit times, from other experiments, alphafix
    clean_6x_tau =  clean_6x_plot['Time (ms)']
    clean_7x_tau =  clean_7x_plot['Time (ms)']
    clean_8x_tau =  clean_8x_plot['Time (ms)']
    clean_9x_tau =  clean_9x_plot['Time (ms)']
    clean_10x_tau = clean_10x_plot['Time (ms)']
    
    # clean transit times, from other experiments, alphafloat
    clean_6y_tau =  clean_6y_plot['Time (ms)']
    clean_7y_tau =  clean_7y_plot['Time (ms)']
    clean_8y_tau =  clean_8y_plot['Time (ms)']
    clean_9y_tau =  clean_9y_plot['Time (ms)']
    clean_10y_tau = clean_10y_plot['Time (ms)']
    
    # clean correlations, alphafix
    clean_1ax_corr = clean_1ax_plot.iloc[:, 1:7:2]
    clean_1bx_corr = clean_1bx_plot.iloc[:, 1:7:2]
    clean_2ax_corr = clean_2ax_plot.iloc[:, 1:7:2]
    clean_2bx_corr = clean_2bx_plot.iloc[:, 1:7:2]
    clean_3ax_corr = clean_3ax_plot.iloc[:, 1:7:2]
    clean_3bx_corr = clean_3bx_plot.iloc[:, 1:7:2]
    clean_4x_corr =  clean_4x_plot.iloc[:, 1:7:2]
    clean_5x_corr =  clean_5x_plot.iloc[:, 1:7:2]
    
    # clean correlations, alphafloat
    clean_1ay_corr = clean_1ay_plot.iloc[:, 1:7:2]
    clean_1by_corr = clean_1by_plot.iloc[:, 1:7:2]
    clean_2ay_corr = clean_2ay_plot.iloc[:, 1:7:2]
    clean_2by_corr = clean_2by_plot.iloc[:, 1:7:2]
    clean_3ay_corr = clean_3ay_plot.iloc[:, 1:7:2]
    clean_3by_corr = clean_3by_plot.iloc[:, 1:7:2]
    clean_4y_corr =  clean_4y_plot.iloc[:, 1:7:2]
    clean_5y_corr =  clean_5y_plot.iloc[:, 1:7:2]
    
    # clean correlations, from other experiments, alphafix
    clean_6x_corr =  clean_6x_plot.iloc[:, 1:7:2]
    clean_7x_corr =  clean_7x_plot.iloc[:, 1:7:2]
    clean_8x_corr =  clean_8x_plot.iloc[:, 1:7:2]
    clean_9x_corr =  clean_9x_plot.iloc[:, 1:7:2]
    clean_10x_corr = clean_10x_plot.iloc[:, 1:7:2]
    
    # clean correlations, from other experiments, alphafloat
    clean_6y_corr =  clean_6y_plot.iloc[:, 1:7:2]
    clean_7y_corr =  clean_7y_plot.iloc[:, 1:7:2]
    clean_8y_corr =  clean_8y_plot.iloc[:, 1:7:2]
    clean_9y_corr =  clean_9y_plot.iloc[:, 1:7:2]
    clean_10y_corr = clean_10y_plot.iloc[:, 1:7:2]
    
    # clean fits, alphafix
    clean_1ax_fit = clean_1ax_plot.iloc[:, 2:7:2]
    clean_1bx_fit = clean_1bx_plot.iloc[:, 2:7:2]
    clean_2ax_fit = clean_2ax_plot.iloc[:, 2:7:2]
    clean_2bx_fit = clean_2bx_plot.iloc[:, 2:7:2]
    clean_3ax_fit = clean_3ax_plot.iloc[:, 2:7:2]
    clean_3bx_fit = clean_3bx_plot.iloc[:, 2:7:2]
    clean_4x_fit =  clean_4x_plot.iloc[:, 2:7:2]
    clean_5x_fit =  clean_5x_plot.iloc[:, 2:7:2]
    
    # clean fits, alphafloat
    clean_1ay_fit = clean_1ay_plot.iloc[:, 2:7:2]
    clean_1by_fit = clean_1by_plot.iloc[:, 2:7:2]
    clean_2ay_fit = clean_2ay_plot.iloc[:, 2:7:2]
    clean_2by_fit = clean_2by_plot.iloc[:, 2:7:2]
    clean_3ay_fit = clean_3ay_plot.iloc[:, 2:7:2]
    clean_3by_fit = clean_3by_plot.iloc[:, 2:7:2]
    clean_4y_fit =  clean_4y_plot.iloc[:, 2:7:2]
    clean_5y_fit =  clean_5y_plot.iloc[:, 2:7:2]
    
    # clean fits, from other experiments, alphafix
    clean_6x_fit =  clean_6x_plot.iloc[:, 2:7:2]
    clean_7x_fit =  clean_7x_plot.iloc[:, 2:7:2]
    clean_8x_fit =  clean_8x_plot.iloc[:, 2:7:2]
    clean_9x_fit =  clean_9x_plot.iloc[:, 2:7:2]
    clean_10x_fit = clean_10x_plot.iloc[:, 2:7:2]
    
    # clean fits, from other experiments, alphafloat
    clean_6y_fit =  clean_6y_plot.iloc[:, 2:7:2]
    clean_7y_fit =  clean_7y_plot.iloc[:, 2:7:2]
    clean_8y_fit =  clean_8y_plot.iloc[:, 2:7:2]
    clean_9y_fit =  clean_9y_plot.iloc[:, 2:7:2]
    clean_10y_fit = clean_10y_plot.iloc[:, 2:7:2]
    
    # dirty plots, alphafix
    dirty_1ax_plot = pd.read_csv(dirty_1ax_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    dirty_1bx_plot = pd.read_csv(dirty_1bx_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    dirty_2ax_plot = pd.read_csv(dirty_2ax_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    dirty_2bx_plot = pd.read_csv(dirty_2bx_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    dirty_3ax_plot = pd.read_csv(dirty_3ax_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    dirty_3bx_plot = pd.read_csv(dirty_3bx_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    dirty_4x_plot =  pd.read_csv(dirty_4x_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    dirty_5x_plot =  pd.read_csv(dirty_5x_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    
    # dirty plots, alphafloat
    dirty_1ay_plot = pd.read_csv(dirty_1ay_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    dirty_1by_plot = pd.read_csv(dirty_1by_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    dirty_2ay_plot = pd.read_csv(dirty_2ay_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    dirty_2by_plot = pd.read_csv(dirty_2by_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    dirty_3ay_plot = pd.read_csv(dirty_3ay_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    dirty_3by_plot = pd.read_csv(dirty_3by_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    dirty_4y_plot =  pd.read_csv(dirty_4y_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    dirty_5y_plot =  pd.read_csv(dirty_5y_plot_path, sep=',', na_values=' ').drop('Unnamed: 107', axis=1)
    
    # dirty plots, from other experiments, alphafix
    dirty_6x_plot =  pd.read_csv(dirty_6x_plot_path, sep=',', na_values=' ').drop('Unnamed: 347', axis=1)
    dirty_7x_plot =  pd.read_csv(dirty_7x_plot_path, sep=',', na_values=' ').drop('Unnamed: 159', axis=1)
    dirty_8x_plot =  pd.read_csv(dirty_8x_plot_path, sep=',', na_values=' ').drop('Unnamed: 159', axis=1)
    dirty_9x_plot =  pd.read_csv(dirty_9x_plot_path, sep=',', na_values=' ').drop('Unnamed: 159', axis=1)
    dirty_10x_plot = pd.read_csv(dirty_10x_plot_path, sep=',', na_values=' ').drop('Unnamed: 159', axis=1)
    
    # dirty plots, from other experiments, alphafloat
    dirty_6y_plot =  pd.read_csv(dirty_6y_plot_path, sep=',', na_values=' ').drop('Unnamed: 347', axis=1)
    dirty_7y_plot =  pd.read_csv(dirty_7y_plot_path, sep=',', na_values=' ').drop('Unnamed: 159', axis=1)
    dirty_8y_plot =  pd.read_csv(dirty_8y_plot_path, sep=',', na_values=' ').drop('Unnamed: 159', axis=1)
    dirty_9y_plot =  pd.read_csv(dirty_9y_plot_path, sep=',', na_values=' ').drop('Unnamed: 159', axis=1)
    dirty_10y_plot = pd.read_csv(dirty_10y_plot_path, sep=',', na_values=' ').drop('Unnamed: 159', axis=1)
    
    # dirty transit times, alphafix
    dirty_1ax_tau = dirty_1ax_plot['Time (ms)']
    dirty_1bx_tau = dirty_1bx_plot['Time (ms)']
    dirty_2ax_tau = dirty_2ax_plot['Time (ms)']
    dirty_2bx_tau = dirty_2bx_plot['Time (ms)']
    dirty_3ax_tau = dirty_3ax_plot['Time (ms)']
    dirty_3bx_tau = dirty_3bx_plot['Time (ms)']
    dirty_4x_tau =  dirty_4x_plot['Time (ms)']
    dirty_5x_tau =  dirty_5x_plot['Time (ms)']
    
    # dirty transit times, alphafloat
    dirty_1ay_tau = dirty_1ay_plot['Time (ms)']
    dirty_1by_tau = dirty_1by_plot['Time (ms)']
    dirty_2ay_tau = dirty_2ay_plot['Time (ms)']
    dirty_2by_tau = dirty_2by_plot['Time (ms)']
    dirty_3ay_tau = dirty_3ay_plot['Time (ms)']
    dirty_3by_tau = dirty_3by_plot['Time (ms)']
    dirty_4y_tau =  dirty_4y_plot['Time (ms)']
    dirty_5y_tau =  dirty_5y_plot['Time (ms)']
    
    # dirty transit times, from other experiments, alphafix
    dirty_6x_tau =  dirty_6x_plot['Time (ms)']
    dirty_7x_tau =  dirty_7x_plot['Time (ms)']
    dirty_8x_tau =  dirty_8x_plot['Time (ms)']
    dirty_9x_tau =  dirty_9x_plot['Time (ms)']
    dirty_10x_tau = dirty_10x_plot['Time (ms)']
    
    # dirty transit times, from other experiments, alphafloat
    dirty_6y_tau =  dirty_6y_plot['Time (ms)']
    dirty_7y_tau =  dirty_7y_plot['Time (ms)']
    dirty_8y_tau =  dirty_8y_plot['Time (ms)']
    dirty_9y_tau =  dirty_9y_plot['Time (ms)']
    dirty_10y_tau = dirty_10y_plot['Time (ms)']
    
    # dirty correlations, alphafix
    dirty_1ax_corr = dirty_1ax_plot.iloc[:, 1:7:2]
    dirty_1bx_corr = dirty_1bx_plot.iloc[:, 1:7:2]
    dirty_2ax_corr = dirty_2ax_plot.iloc[:, 1:7:2]
    dirty_2bx_corr = dirty_2bx_plot.iloc[:, 1:7:2]
    dirty_3ax_corr = dirty_3ax_plot.iloc[:, 1:7:2]
    dirty_3bx_corr = dirty_3bx_plot.iloc[:, 1:7:2]
    dirty_4x_corr =  dirty_4x_plot.iloc[:, 1:7:2]
    dirty_5x_corr =  dirty_5x_plot.iloc[:, 1:7:2]
    
    # dirty correlations, alphafloat
    dirty_1ay_corr = dirty_1ay_plot.iloc[:, 1:7:2]
    dirty_1by_corr = dirty_1by_plot.iloc[:, 1:7:2]
    dirty_2ay_corr = dirty_2ay_plot.iloc[:, 1:7:2]
    dirty_2by_corr = dirty_2by_plot.iloc[:, 1:7:2]
    dirty_3ay_corr = dirty_3ay_plot.iloc[:, 1:7:2]
    dirty_3by_corr = dirty_3by_plot.iloc[:, 1:7:2]
    dirty_4y_corr =  dirty_4y_plot.iloc[:, 1:7:2]
    dirty_5y_corr =  dirty_5y_plot.iloc[:, 1:7:2]
    
    # dirty correlations, from other experiments, alphafix
    dirty_6x_corr =  dirty_6x_plot.iloc[:, 1:7:2]
    dirty_7x_corr =  dirty_7x_plot.iloc[:, 1:7:2]
    dirty_8x_corr =  dirty_8x_plot.iloc[:, 1:7:2]
    dirty_9x_corr =  dirty_9x_plot.iloc[:, 1:7:2]
    dirty_10x_corr = dirty_10x_plot.iloc[:, 1:7:2]
    
    # dirty correlations, from other experiments, alphafloat
    dirty_6y_corr =  dirty_6y_plot.iloc[:, 1:7:2]
    dirty_7y_corr =  dirty_7y_plot.iloc[:, 1:7:2]
    dirty_8y_corr =  dirty_8y_plot.iloc[:, 1:7:2]
    dirty_9y_corr =  dirty_9y_plot.iloc[:, 1:7:2]
    dirty_10y_corr = dirty_10y_plot.iloc[:, 1:7:2]
    
    # dirty fits, alphafix
    dirty_1ax_fit = dirty_1ax_plot.iloc[:, 2:7:2]
    dirty_1bx_fit = dirty_1bx_plot.iloc[:, 2:7:2]
    dirty_2ax_fit = dirty_2ax_plot.iloc[:, 2:7:2]
    dirty_2bx_fit = dirty_2bx_plot.iloc[:, 2:7:2]
    dirty_3ax_fit = dirty_3ax_plot.iloc[:, 2:7:2]
    dirty_3bx_fit = dirty_3bx_plot.iloc[:, 2:7:2]
    dirty_4x_fit =  dirty_4x_plot.iloc[:, 2:7:2]
    dirty_5x_fit =  dirty_5x_plot.iloc[:, 2:7:2]
    
    # dirty fits, alphafloat
    dirty_1ay_fit = dirty_1ay_plot.iloc[:, 2:7:2]
    dirty_1by_fit = dirty_1by_plot.iloc[:, 2:7:2]
    dirty_2ay_fit = dirty_2ay_plot.iloc[:, 2:7:2]
    dirty_2by_fit = dirty_2by_plot.iloc[:, 2:7:2]
    dirty_3ay_fit = dirty_3ay_plot.iloc[:, 2:7:2]
    dirty_3by_fit = dirty_3by_plot.iloc[:, 2:7:2]
    dirty_4y_fit =  dirty_4y_plot.iloc[:, 2:7:2]
    dirty_5y_fit =  dirty_5y_plot.iloc[:, 2:7:2]
    
    # dirty fits, from other experiments, alphafix
    dirty_6x_fit =  dirty_6x_plot.iloc[:, 2:7:2]
    dirty_7x_fit =  dirty_7x_plot.iloc[:, 2:7:2]
    dirty_8x_fit =  dirty_8x_plot.iloc[:, 2:7:2]
    dirty_9x_fit =  dirty_9x_plot.iloc[:, 2:7:2]
    dirty_10x_fit = dirty_10x_plot.iloc[:, 2:7:2]
    
    # dirty fits, from other experiments, alphafloat
    dirty_6y_fit =  dirty_6y_plot.iloc[:, 2:7:2]
    dirty_7y_fit =  dirty_7y_plot.iloc[:, 2:7:2]
    dirty_8y_fit =  dirty_8y_plot.iloc[:, 2:7:2]
    dirty_9y_fit =  dirty_9y_plot.iloc[:, 2:7:2]
    dirty_10y_fit = dirty_10y_plot.iloc[:, 2:7:2]
    
    all_plot_clean = [{'tttr2xfcs' : {'alpha=1' : {'tau': clean_1ax_tau, 'corr': clean_1ax_corr, 'fit': clean_1ax_fit},
                                'alpha=float' : {'tau': clean_1ay_tau, 'corr': clean_1ay_corr, 'fit': clean_1ay_fit}},
                 'multipletau': {'alpha=1' : {'tau': clean_1bx_tau, 'corr': clean_1bx_corr, 'fit': clean_1bx_fit},
                                 'alpha=float': {'tau': clean_1by_tau, 'corr': clean_1by_corr, 'fit': clean_1by_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': clean_2ax_tau, 'corr': clean_2ax_corr, 'fit': clean_2ax_fit},
                                'alpha=float' : {'tau': clean_2ay_tau, 'corr': clean_2ay_corr, 'fit': clean_2ay_fit}},
                 'multipletau': {'alpha=1' : {'tau': clean_2bx_tau, 'corr': clean_2bx_corr, 'fit': clean_2bx_fit},
                                 'alpha=float': {'tau': clean_2by_tau, 'corr': clean_2by_corr, 'fit': clean_2by_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': clean_3ax_tau, 'corr': clean_3ax_corr, 'fit': clean_3ax_fit},
                                'alpha=float' : {'tau': clean_3ay_tau, 'corr': clean_3ay_corr, 'fit': clean_3ay_fit}},
                 'multipletau': {'alpha=1' : {'tau': clean_3bx_tau, 'corr': clean_3bx_corr, 'fit': clean_3bx_fit},
                                 'alpha=float': {'tau': clean_3by_tau, 'corr': clean_3by_corr, 'fit': clean_3by_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': clean_4x_tau, 'corr': clean_4x_corr, 'fit': clean_4x_fit},
                                'alpha=float' : {'tau': clean_4y_tau, 'corr': clean_4y_corr, 'fit': clean_4y_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': clean_5x_tau, 'corr': clean_5x_corr, 'fit': clean_5x_fit},
                                'alpha=float' : {'tau': clean_5y_tau, 'corr': clean_5y_corr, 'fit': clean_5y_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': clean_6x_tau, 'corr': clean_6x_corr, 'fit': clean_6x_fit},
                                'alpha=float' : {'tau': clean_6y_tau, 'corr': clean_6y_corr, 'fit': clean_6y_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': clean_7x_tau, 'corr': clean_7x_corr, 'fit': clean_7x_fit},
                                'alpha=float' : {'tau': clean_7y_tau, 'corr': clean_7y_corr, 'fit': clean_7y_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': clean_8x_tau, 'corr': clean_8x_corr, 'fit': clean_8x_fit},
                                'alpha=float' : {'tau': clean_8y_tau, 'corr': clean_8y_corr, 'fit': clean_8y_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': clean_9x_tau, 'corr': clean_9x_corr, 'fit': clean_9x_fit},
                                'alpha=float' : {'tau': clean_9y_tau, 'corr': clean_9y_corr, 'fit': clean_9y_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': clean_10x_tau, 'corr': clean_10x_corr, 'fit': clean_10x_fit},
                                'alpha=float' : {'tau': clean_10y_tau, 'corr': clean_10y_corr, 'fit': clean_10y_fit}}}]
    all_plot_dirty = [{'tttr2xfcs' : {'alpha=1' : {'tau': dirty_1ax_tau, 'corr': dirty_1ax_corr, 'fit': dirty_1ax_fit},
                                'alpha=float' : {'tau': dirty_1ay_tau, 'corr': dirty_1ay_corr, 'fit': dirty_1ay_fit}},
                 'multipletau': {'alpha=1' : {'tau': dirty_1bx_tau, 'corr': dirty_1bx_corr, 'fit': dirty_1bx_fit},
                                 'alpha=float': {'tau': dirty_1by_tau, 'corr': dirty_1by_corr, 'fit': dirty_1by_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': dirty_2ax_tau, 'corr': dirty_2ax_corr, 'fit': dirty_2ax_fit},
                                'alpha=float' : {'tau': dirty_2ay_tau, 'corr': dirty_2ay_corr, 'fit': dirty_2ay_fit}},
                 'multipletau': {'alpha=1' : {'tau': dirty_2bx_tau, 'corr': dirty_2bx_corr, 'fit': dirty_2bx_fit},
                                 'alpha=float': {'tau': dirty_2by_tau, 'corr': dirty_2by_corr, 'fit': dirty_2by_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': dirty_3ax_tau, 'corr': dirty_3ax_corr, 'fit': dirty_3ax_fit},
                                'alpha=float' : {'tau': dirty_3ay_tau, 'corr': dirty_3ay_corr, 'fit': dirty_3ay_fit}},
                 'multipletau': {'alpha=1' : {'tau': dirty_3bx_tau, 'corr': dirty_3bx_corr, 'fit': dirty_3bx_fit},
                                 'alpha=float': {'tau': dirty_3by_tau, 'corr': dirty_3by_corr, 'fit': dirty_3by_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': dirty_4x_tau, 'corr': dirty_4x_corr, 'fit': dirty_4x_fit},
                                'alpha=float' : {'tau': dirty_4y_tau, 'corr': dirty_4y_corr, 'fit': dirty_4y_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': dirty_5x_tau, 'corr': dirty_5x_corr, 'fit': dirty_5x_fit},
                                'alpha=float' : {'tau': dirty_5y_tau, 'corr': dirty_5y_corr, 'fit': dirty_5y_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': dirty_6x_tau, 'corr': dirty_6x_corr, 'fit': dirty_6x_fit},
                                'alpha=float' : {'tau': dirty_6y_tau, 'corr': dirty_6y_corr, 'fit': dirty_6y_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': dirty_7x_tau, 'corr': dirty_7x_corr, 'fit': dirty_7x_fit},
                                'alpha=float' : {'tau': dirty_7y_tau, 'corr': dirty_7y_corr, 'fit': dirty_7y_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': dirty_8x_tau, 'corr': dirty_8x_corr, 'fit': dirty_8x_fit},
                                'alpha=float' : {'tau': dirty_8y_tau, 'corr': dirty_8y_corr, 'fit': dirty_8y_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': dirty_9x_tau, 'corr': dirty_9x_corr, 'fit': dirty_9x_fit},
                                'alpha=float' : {'tau': dirty_9y_tau, 'corr': dirty_9y_corr, 'fit': dirty_9y_fit}}},
                {'tttr2xfcs' : {'alpha=1' : {'tau': dirty_10x_tau, 'corr': dirty_10x_corr, 'fit': dirty_10x_fit},
                                'alpha=float' : {'tau': dirty_10y_tau, 'corr': dirty_10y_corr, 'fit': dirty_10y_fit}}}]
    index = ['no correction', 'delete', 'delete and shift', 'w=1-pred', 'w=random', 'w=0', 'w=0.2', 'w=0.4', 'w=0.6', 'w=0.8']
    
    
  • now, let's plot correlations and fits for all correction methods:
    fig = plt.figure(figsize=(12, 40))
    rows = len(index) + 3
    counter = 1
    palette_clean = [sns.color_palette()[0], sns.color_palette()[1], sns.color_palette()[2]]
    palette_dirty = [sns.color_palette()[3], sns.color_palette()[4], sns.color_palette()[5]]
    for i, correction_clean, correction_dirty in zip(index, all_plot_clean, all_plot_dirty):
        for (method_clean, fitplot_clean), (method_dirty, fitplot_dirty) in zip(correction_clean.items(), correction_dirty.items()):
            for (fit_clean, plots_clean), (fit_dirty, plots_dirty) in zip(fitplot_clean.items(), fitplot_dirty.items()):
                # clean
                if fit_clean == 'alpha=1':
                    fit_color = 'r'
                else:
                    fit_color= 'b'
                ax = plt.subplot(rows, 4, counter)
                ax.set_prop_cycle(color=palette_clean)
                lines1 = plt.semilogx(plots_clean['tau'], plots_clean['corr'], '.')
                plt.semilogx(plots_clean['tau'], plots_clean['fit'], f'{fit_color}-')
                xmin = plots_clean['fit'].dropna().index[0]
                xmax = plots_clean['fit'].dropna().index[-1]
                xlims = [plots_clean['tau'][xmin] - 0.5*plots_clean['tau'][xmin], plots_clean['tau'][xmax]+0.5*plots_clean['tau'][xmax]]
                ylims = [np.min(np.min(plots_clean['fit']))-0.01, np.max(np.max(plots_clean['fit']))+0.01]
                plt.setp(ax, xlim=xlims, ylim=ylims, title=f'clean | {i}\n{method_clean} | {fit_clean}',
                         xlabel=r'$\tau$ (ms)', ylabel=r'Correlation G($\tau$)')
                counter += 1
                # dirty
                ax = plt.subplot(rows, 4, counter)
                ax.set_prop_cycle(color=palette_dirty)
                lines2 = plt.semilogx(plots_dirty['tau'], plots_dirty['corr'], '.')
                plt.semilogx(plots_dirty['tau'], plots_dirty['fit'], f'{fit_color}-')
                xmin = plots_dirty['fit'].dropna().index[0]
                xmax = plots_dirty['fit'].dropna().index[-1]
                xlims = [plots_dirty['tau'][xmin] - 0.5*plots_dirty['tau'][xmin], plots_dirty['tau'][xmax]+0.5*plots_dirty['tau'][xmax]]
                ylims = [np.min(np.min(plots_dirty['fit']))-0.01, np.max(np.max(plots_dirty['fit']))+0.01]
                plt.setp(ax, xlim=xlims, ylim=ylims, title=f'dirty | {i}\n{method_dirty} | {fit_dirty}',
                         xlabel=r'$\tau$ (ms)', ylabel=r'Correlation G($\tau$)')
                if counter < 3:
                    handles = lines1 + lines2
                    fig.legend(handles, ['20 nM AF48816_T182s_1', '20 nM AF48822_T254s_1', '20 nM AF48826_T302s_1',
                                         'DiO LUV 10uM in 20 nM AF4884_T39s_1', 'DiO LUV 10uM in 20 nM AF4887_T75s_1',
                                         'DiO LUV 10uM in 20 nM AF4889_T106s_1'],
                               loc=9, ncol=2, bbox_to_anchor=(0.5, 0.98))
                counter += 1
    
    fig.suptitle('AlexaFluor488 (clean) vs AlexaFluor488 + Dio LUVs (dirty)\n- all correlations and fits -',
                 size=20, y=1)
    plt.tight_layout(rect=(0, 0, 1, 0.98))
    

analysis3-all-plot.png

2.5.11 Analysis 4: show examples of all experimental traces

:header-args:jupyter-python: :session /jpy:localhost#8889:a37e524a-8134-4d8f-b24a-367acaf1bdd3

  • to interprete the correlations correctly, let's plot the underlying experimental data.
    %cd ~/Programme/drmed-collections/data-from-Eggeling-group/brightbursts
    
    /home/lex/Programme/drmed-collections/data-from-Eggeling-group/brightbursts
    
    %cd /beegfs/ye53nis/drmed-git
    
    /beegfs/ye53nis/drmed-git
    
  • load modules
    import logging
    import os
    import sys
    
    import matplotlib.pyplot as plt
    import numpy as np
    import pandas as pd
    import seaborn as sns
    
    from pathlib import Path
    from pprint import pprint
    from tensorflow.keras.optimizers import Adam
    from mlflow.keras import load_model
    
    FLUOTRACIFY_PATH = '/beegfs/ye53nis/drmed-git/src/'
    sys.path.append(FLUOTRACIFY_PATH)
    from fluotracify.applications import corr_fit_object as cfo
    from fluotracify.training import build_model as bm
    
    logging.basicConfig(filename="/beegfs/ye53nis/drmed-git/data/exp-220120-correlate-ptu/2022-02-15_experiment-overview/exps.log",
                        filemode='w', format='%(asctime)s - %(message)s',
                        force=True,
                        level=logging.DEBUG)
    # sns.set_theme(style="whitegrid")
    sns.set()
    
    2022-02-17 19:45:03.282584: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2022-02-17 19:45:03.282645: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    
  • first, we prepare our correction functions as we did before
    class ParameterClass():
        """Stores parameters for correlation """
        def __init__(self):
            # Where the data is stored.
            self.data = []
            self.objectRef = []
            self.subObjectRef = []
            self.colors = ['blue', 'green', 'red', 'cyan', 'magenta',
                           'yellow', 'black']
            self.numOfLoaded = 0
            # very fast from Ncasc ~ 14 onwards
            self.NcascStart = 0
            self.NcascEnd = 30  # 25
            self.Nsub = 6  # 6
            self.photonLifetimeBin = 10  # used for photon decay
            self.photonCountBin = 1  # used for time series
    
    data_path = Path("/beegfs/ye53nis/data")
    output_path = "/beegfs/ye53nis/drmed-git/data/exp-220120-correlate-ptu/2022-02-15_experiment-overview"
    logged_model = 'file:///beegfs/ye53nis/drmed-git/data/mlruns/8/00f2635d9fa2463c9a066722163405be/artifacts/model'
    par_obj = ParameterClass()
    
    loaded_model = load_model(logged_model, compile=False)
    loaded_model.compile(loss=bm.binary_ce_dice_loss(),
                         optimizer=Adam(),
                         metrics = bm.unet_metrics([0.1, 0.3, 0.5, 0.7, 0.9]))
    bm.prepare_model(loaded_model, [2**14, 2**13, 2**12, 2**15, 2**13])
    
    2022-02-17 19:45:12.370211: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2022-02-17 19:45:12.370284: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
    2022-02-17 19:45:12.370324: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node305): /proc/driver/nvidia/version does not exist
    2022-02-17 19:45:12.370974: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    
  • look at first bright cluster artifacts folder: 1911DD_alexafluor+LUVs
    • clean = 20nM Alexa Fluor 488, a green-fluorescent dye, pH-insensitive, relatively bright and photostable. We expect a diffusion constant of 350-450 um2/s = 25-32us transit times
    • dirty = 20nM Alexa Fluor 488 + 10uM DiO LUV → 10uM of lipids, labelled with the dye DiO (also green) in the membrane. we expect there to be the fast fraction with 25-32us transit times as above and a slow fraction with a diffusion constant of 5 um2/s = 2250us transit time
      • LUV preparation method: extrusion (LUVET), detergent dialysis (DOV), reverse evaporation (REV) or ethanol injection? → extrusion through 100nm filter
      • large unilamellar vesicles prepared from large multilamellar vesicles (LMVs). Here they are 100nm in diameter or larger. Are stable on storage.
      • where is the dye?? In the membrane or inside the vesicle or both?? → in the membrane. We don't expect to see the diffusion of DiO in the membrane, but rather the diffusion of the LUV itself, because the one in the membrane can not go in and out of the excitation volume
      • which fitting methods should we choose? → we should do 3D fitting with equation 1B, fix the AR1 (propert of microscope setup which converts between txy and tz) between 4.5 and 6, and alpha at 1 (no anomalous diffusion). We can also try 2 component fitting!
    • let's plot traces, predictions, and correlations:
      path_clean = data_path / '1911DD_alexafluor488+LUVs/clean_subsample/'
      path_dirty = data_path / '1911DD_alexafluor488+LUVs/dirty_subsample/'
      files_clean = [path_clean / f for f in os.listdir(path_clean) if f.endswith('.ptu')]
      files_dirty = [path_dirty / f for f in os.listdir(path_dirty) if f.endswith('.ptu')]
      traces = pd.DataFrame()
      predtraces = pd.DataFrame()
      preds = pd.DataFrame()
      corrs = pd.DataFrame()
      mt_bin = 1e-3
      
      
      for myfile in (files_dirty):
          ptufile = cfo.PicoObject(myfile, par_obj)
          ptufile.predictTimeSeries(model=loaded_model,
                                    scaler='minmax')
          mt_name = f'us_{ptufile.name}'
          ptufile.getTimeSeries(timeseries_name=mt_name,
                                photonCountBin=mt_bin)
          ptufile.getPhotonCountingStats(name=mt_name)
          ptufile.predictTimeSeries(model=loaded_model,
                                    scaler='minmax',
                                    name=mt_name)
          for key in list(ptufile.trueTimeArr.keys()):
              ptufile.get_autocorrelation(method='tttr2xfcs', name=key)
          for key in list(ptufile.timeSeries.keys()):
              for k, i in ptufile.timeSeries[key].items():
                  if "PREPRO" in k:
                      if "1.0" in k:
                          predtraces = pd.concat([predtraces, pd.DataFrame(i, columns=[f'{key}_{k}'])],
                                                 axis=1)
                  elif "1.0" in k:
                      traces = pd.concat([traces, pd.DataFrame(i, columns=[f'{key}_{k}'])],
                                         axis=1)
                      preds = pd.concat([preds, pd.DataFrame(data=ptufile.predictions[key][k],
                                                             columns=[f'{key}_{k}'])], axis=1)
                  elif "0.001" in k:
                      ptufile.get_autocorrelation(method='multipletau', name=(key, k))
      
          for m in ['multipletau', 'tttr2xfcs', 'tttr2xfcs_with_weights']:
              if m in list(ptufile.autoNorm.keys()):
                  for key, item in list(ptufile.autoNorm[m].items()):
                      ptufile.save_autocorrelation(name=key, method=m,
                                                   output_path=output_path)
      predtraces.to_csv(Path(output_path) / 'alexadirty_predtraces.csv')
      traces.to_csv(Path(output_path) / 'alexadirty_traces.csv')
      preds.to_csv(Path(output_path) / 'alexadirty_preds.csv')
      
      /home/lex/Programme/miniconda3/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      /home/lex/Programme/miniconda3/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      /home/lex/Programme/miniconda3/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      
    • now I fitted the example traces using https://dwaithe.github.io/FCSfitJS/. Because of disk space issues, the .ptu files above are not available in this git repository. Rather, I saved out the traces, prediction traces and predictions and loaded them in, together with the correlations and fits:
      fit_dirty_mt = pd.read_csv(Path(output_path) / '1911DD_alexafluor488+LUVs/dirty/alexadirty_multipletau_plot.csv',
                                     index_col=0, usecols=[0, 2, 4, 6], na_values=' ')
      fit_dirty_tt = pd.read_csv(Path(output_path) / '1911DD_alexafluor488+LUVs/dirty/alexadirty_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 2, 4, 6], na_values=' ')
      fit_clean_mt = pd.read_csv(Path(output_path) / '1911DD_alexafluor488+LUVs/clean/alexaclean_multipletau_plot.csv'
                                     , index_col=0, usecols=[0, 2, 4, 6], na_values=' ')
      fit_clean_tt = pd.read_csv(Path(output_path) / '1911DD_alexafluor488+LUVs/clean/alexaclean_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 2, 4, 6], na_values=' ')
      corr_dirty_mt = pd.read_csv(Path(output_path) / '1911DD_alexafluor488+LUVs/dirty/alexadirty_multipletau_plot.csv',
                                     index_col=0, usecols=[0, 1, 3, 5], na_values=' ')
      corr_dirty_tt = pd.read_csv(Path(output_path) / '1911DD_alexafluor488+LUVs/dirty/alexadirty_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 1, 3, 5], na_values=' ')
      corr_clean_mt = pd.read_csv(Path(output_path) / '1911DD_alexafluor488+LUVs/clean/alexaclean_multipletau_plot.csv'
                                     , index_col=0, usecols=[0, 1, 3, 5], na_values=' ')
      corr_clean_tt = pd.read_csv(Path(output_path) / '1911DD_alexafluor488+LUVs/clean/alexaclean_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 1, 3, 5], na_values=' ')
      
      preds_dirty = pd.read_csv(Path(output_path) / '1911DD_alexafluor488+LUVs/dirty/alexadirty_preds.csv', index_col=0)
      predtraces_dirty = pd.read_csv(Path(output_path) / '1911DD_alexafluor488+LUVs/dirty/alexadirty_predtraces.csv', index_col=0)
      traces_dirty = pd.read_csv(Path(output_path) / '1911DD_alexafluor488+LUVs/dirty/alexadirty_traces.csv', index_col=0)
      preds_clean = pd.read_csv(Path(output_path) / '1911DD_alexafluor488+LUVs/clean/alexaclean_preds.csv', index_col=0)
      predtraces_clean = pd.read_csv(Path(output_path) / '1911DD_alexafluor488+LUVs/clean/alexaclean_predtraces.csv', index_col=0)
      traces_clean = pd.read_csv(Path(output_path) / '1911DD_alexafluor488+LUVs/clean/alexaclean_traces.csv', index_col=0)
      fit_clean_mt.columns
      
      Index(['2022-02-02_multipletau_CH2_BIN0dot001_us_20 nM AF4887_T74s_1_correlation-CH2_2 fitted model: ',
             '2022-02-02_multipletau_CH2_BIN0dot001_us_20 nM AF48896_T1142s_1_correlation-CH2_2 fitted model: ',
             '2022-02-02_multipletau_CH2_BIN0dot001_us_20 nM AF48897_T1160s_1_correlation-CH2_2 fitted model: '],
            dtype='object')
      
      plotdata = zip(traces_dirty.items(), traces_clean.items(), predtraces_dirty.items(),
                     predtraces_clean.items(), preds_dirty.items(), preds_clean.items(),
                     corr_dirty_mt.items(), corr_clean_mt.items(), corr_dirty_tt.items(),
                     corr_clean_tt.items(), fit_dirty_mt.items(), fit_clean_mt.items(),
                     fit_dirty_tt.items(), fit_clean_tt.items())
      fig = plt.figure(figsize=(16, 20))
      gs = fig.add_gridspec(12, 4)
      for i, ((td_n, td_v), (tc_n, tc_v), (_, ptd_v), (_, ptc_v), (_, pd_v),
              (_, pc_v), (_, cdm_v), (_, ccm_v), (_, cdt_v), (_, cct_v),
              (_, fdm_v), (_, fcm_v), (_, fdt_v), (_, fct_v)) in enumerate(plotdata):
          print(i)
          ax0 = fig.add_subplot(gs[i*4, 0:2])
          sns.lineplot(ax=ax0, data=(tc_n, tc_v))
          ax1 = fig.add_subplot(gs[i*4, 2:4])
          sns.lineplot(ax=ax1, data=(td_n, td_v))
          ax0.set(xlabel='time steps in ms', ylabel='photons')
          ax1.set(xlabel='time steps in ms', ylabel='photons')
          ax2 = fig.add_subplot(gs[i*4+1, 0:2], sharex=ax0)
          sns.lineplot(ax=ax2, data=ptc_v, color=sns.color_palette()[0], legend=False)
          sns.lineplot(ax=ax2, data=pc_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax3 = fig.add_subplot(gs[i*4+1, 2:4], sharex=ax1)
          sns.lineplot(ax=ax3, data=ptd_v, legend=False)
          sns.lineplot(ax=ax3, data=pd_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax2.set(xlabel='time steps in ms', ylabel='probability of artifact')
          ax3.set(xlabel='time steps in ms', ylabel='probability of artifact')
          ax4 = fig.add_subplot(gs[i*4+2:i*4+4, 0], title='multipletau w/ us bin')
          sns.lineplot(ax=ax4, data=ccm_v, marker='.', legend=False)
          sns.lineplot(ax=ax4, data=fcm_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax5 = fig.add_subplot(gs[i*4+2:i*4+4, 1], title='tttr2xfcs on arrival times')
          sns.lineplot(ax=ax5, data=cct_v, marker='.', legend=False)
          sns.lineplot(ax=ax5, data=fct_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax4.set(xscale='log', xlabel='tau in ms', ylabel='Correlation G(tau)')
          ax5.set(xscale='log', xlabel='tau in ms', ylabel='Correlation G(tau)')
          ax6 = fig.add_subplot(gs[i*4+2:i*4+4, 2], title='multipletau w/ us bin', sharey=ax4)
          sns.lineplot(ax=ax6, data=cdm_v, marker='.', legend=False)
          sns.lineplot(ax=ax6, data=fdm_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax7 = fig.add_subplot(gs[i*4+2:i*4+4, 3], title='tttr2xfcs on arrival times', sharey=ax5)
          sns.lineplot(ax=ax7, data=cdt_v, marker='.', legend=False)
          sns.lineplot(ax=ax7, data=fdt_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax6.set(xscale='log', xlabel='tau in ms', ylabel='Correlation G(tau)')
          ax7.set(xscale='log', xlabel='tau in ms', ylabel='Correlation G(tau)')
      fig.suptitle('AlexaFluor488 (left) vs AF488 + DiO-LUVs (right)', size=20)
      gs.tight_layout(fig, rect=[0, 0, 1, 0.99])
      
      
    • the results: analysis4-1911DD-alexa-luvs.png
  • look at second bright cluster artifacts folder: 191111_Pex5_1
    • clean = Homo sapiens PEX5, labelled with eGFP
    • dirty = Trypanosoma brucei PEX5, labelled with eGFP
    • both measurements have 2 channels! → Pablo said that is because he didn't know the microscope well → I can just use the channel with the higher count rates.
    • let's plot traces, predictions, and correlations (see Details blocks):
      path_clean = data_path / '191111_Pex5_1/clean_3of10'
      path_dirty = data_path / '191111_Pex5_1/Tb-PEX5-eGFP_5uW_0.5ugml3_3of177'
      files_clean = [path_clean / f for f in os.listdir(path_clean) if f.endswith('.ptu')]
      files_dirty = [path_dirty / f for f in os.listdir(path_dirty) if f.endswith('.ptu')]
      traces = pd.DataFrame()
      predtraces = pd.DataFrame()
      preds = pd.DataFrame()
      corrs = pd.DataFrame()
      mt_bin = 1e-3
      
      
      for myfile in (files_dirty):
          ptufile = cfo.PicoObject(myfile, par_obj)
          # ptufile.predictTimeSeries(model=loaded_model,
          #                           scaler='minmax')
          mt_name = f'us_{ptufile.name}'
          ptufile.getTimeSeries(timeseries_name=mt_name,
                                photonCountBin=mt_bin)
          ptufile.getPhotonCountingStats(name=mt_name)
          # ptufile.predictTimeSeries(model=loaded_model,
          #                           scaler='minmax',
          #                           name=mt_name)
          for key in list(ptufile.trueTimeArr.keys()):
              ptufile.get_autocorrelation(method='tttr2xfcs', name=key)
          for key in list(ptufile.timeSeries.keys()):
              for k, i in ptufile.timeSeries[key].items():
                  if "PREPRO" in k:
                      if "1.0" in k:
                          # predtraces = pd.concat([predtraces, pd.DataFrame(i, columns=[f'{key}_{k}'])],
                          #                        axis=1)
                          pass
                  elif "1.0" in k:
                      traces = pd.concat([traces, pd.DataFrame(i, columns=[f'{key}_{k}'])],
                                         axis=1)
                      # preds = pd.concat([preds, pd.DataFrame(data=ptufile.predictions[key][k],
                      #                                        columns=[f'{key}_{k}'])], axis=1)
                  elif "0.001" in k:
                      ptufile.get_autocorrelation(method='multipletau', name=(key, k))
      
          for m in ['multipletau', 'tttr2xfcs', 'tttr2xfcs_with_weights']:
              if m in list(ptufile.autoNorm.keys()):
                  for key, item in list(ptufile.autoNorm[m].items()):
                      ptufile.save_autocorrelation(name=key, method=m,
                                                   output_path=output_path)
      # predtraces.to_csv(Path(output_path) / 'pex2dirty_predtraces.csv')
      traces.to_csv(Path(output_path) / 'pex1dirty_traces.csv')
      # preds.to_csv(Path(output_path) / 'pex2dirty_preds.csv')
      
      /home/lex/Programme/miniconda3/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      
      fit_dirty_mt = pd.read_csv(Path(output_path) / '191111_Pex5_1/dirty/pex1dirty_CH2_multipletau_plot.csv',
                                     index_col=0, usecols=[0, 2, 4, 6], na_values=' ')
      fit_dirty_tt = pd.read_csv(Path(output_path) / '191111_Pex5_1/dirty/pex1dirty_CH2_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 2, 4, 6], na_values=' ')
      fit_clean_mt = pd.read_csv(Path(output_path) / '191111_Pex5_1/clean/pex1clean_CH2_multipletau_plot.csv'
                                     , index_col=0, usecols=[0, 2, 4, 6], na_values=' ')
      fit_clean_tt = pd.read_csv(Path(output_path) / '191111_Pex5_1/clean/pex1clean_CH2_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 2, 4, 6], na_values=' ')
      corr_dirty_mt = pd.read_csv(Path(output_path) / '191111_Pex5_1/dirty/pex1dirty_CH2_multipletau_plot.csv',
                                     index_col=0, usecols=[0, 1, 3, 5], na_values=' ')
      corr_dirty_tt = pd.read_csv(Path(output_path) / '191111_Pex5_1/dirty/pex1dirty_CH2_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 1, 3, 5], na_values=' ')
      corr_clean_mt = pd.read_csv(Path(output_path) / '191111_Pex5_1/clean/pex1clean_CH2_multipletau_plot.csv'
                                     , index_col=0, usecols=[0, 1, 3, 5], na_values=' ')
      corr_clean_tt = pd.read_csv(Path(output_path) / '191111_Pex5_1/clean/pex1clean_CH2_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 1, 3, 5], na_values=' ')
      
      # preds_dirty = pd.read_csv(Path(output_path) / '191111_Pex5_1/dirty/pex1dirty_preds.csv', index_col=0)
      # predtraces_dirty = pd.read_csv(Path(output_path) / '191111_Pex5_1/dirty/pex1dirty_predtraces.csv', index_col=0)
      traces_dirty = pd.read_csv(Path(output_path) / '191111_Pex5_1/dirty/pex1dirty_traces.csv',
                                 index_col=0, usecols=[0, 2, 4, 6])
      # preds_clean = pd.read_csv(Path(output_path) / '191111_Pex5_1/clean/pex1clean_preds.csv', index_col=0)
      # predtraces_clean = pd.read_csv(Path(output_path) / '191111_Pex5_1/clean/pex1clean_predtraces.csv', index_col=0)
      traces_clean = pd.read_csv(Path(output_path) / '191111_Pex5_1/clean/pex1clean_traces.csv',
                                 index_col=0, usecols=[0, 2, 4, 6])
      print(fit_clean_mt.columns)
      print(traces_clean.columns)
      
      Index(['2022-02-03_multipletau_CH2_BIN0dot001_us_2Hs-PEX5-eGFP5_T91s_1_correlation-CH2_2 fitted model: ',
             '2022-02-03_multipletau_CH2_BIN0dot001_us_Hs-PEX5-eGFP1_T0s_1_correlation-CH2_2 fitted model: ',
             '2022-02-03_multipletau_CH2_BIN0dot001_us_Hs-PEX5-eGFP2_T26s_1_correlation-CH2_2 fitted model: '],
            dtype='object')
      Index(['Hs-PEX5-eGFP1_T0s_1_CH2_BIN1.0', '2Hs-PEX5-eGFP5_T91s_1_CH2_BIN1.0',
             'Hs-PEX5-eGFP2_T26s_1_CH2_BIN1.0'],
            dtype='object')
      
      plotdata = zip(traces_dirty.items(), traces_clean.items(),
                     corr_dirty_mt.items(), corr_clean_mt.items(), corr_dirty_tt.items(),
                     corr_clean_tt.items(), fit_dirty_mt.items(), fit_clean_mt.items(),
                     fit_dirty_tt.items(), fit_clean_tt.items())
      fig = plt.figure(figsize=(16, 20))
      gs = fig.add_gridspec(12, 4)
      for i, ((td_n, td_v), (td_n, tc_v), (_, cdm_v), (_, ccm_v), (_, cdt_v), (_, cct_v),
              (_, fdm_v), (_, fcm_v), (_, fdt_v), (_, fct_v)) in enumerate(plotdata):
          print(i)
          ax0 = fig.add_subplot(gs[i*4, 0:2])
          sns.lineplot(ax=ax0, data=(td_n, tc_v), legend='auto')
          ax1 = fig.add_subplot(gs[i*4, 2:4])
          sns.lineplot(ax=ax1, data=(td_n, td_v), legend='auto')
          ax0.set(xlabel='time steps in ms', ylabel='photons')
          ax1.set(xlabel='time steps in ms', ylabel='photons')
          # ax2 = fig.add_subplot(gs[i*4+1, 0:2], sharex=ax0)
          # sns.lineplot(ax=ax2, data=ptc_v, color=sns.color_palette()[0], legend=False)
          # sns.lineplot(ax=ax2, data=pc_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          # ax3 = fig.add_subplot(gs[i*4+1, 2:4], sharex=ax1)
          # sns.lineplot(ax=ax3, data=ptd_v, legend=False)
          # sns.lineplot(ax=ax3, data=pd_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          # ax2.set(xlabel='time steps in ms', ylabel='probability of artifact')
          # ax3.set(xlabel='time steps in ms', ylabel='probability of artifact')
          ax4 = fig.add_subplot(gs[i*4+2:i*4+4, 0], title='multipletau w/ us bin')
          sns.lineplot(ax=ax4, data=ccm_v, marker='.', legend=False)
          sns.lineplot(ax=ax4, data=fcm_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax5 = fig.add_subplot(gs[i*4+2:i*4+4, 1], title='tttr2xfcs on arrival times')
          sns.lineplot(ax=ax5, data=cct_v, marker='.', legend=False)
          sns.lineplot(ax=ax5, data=fct_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax4.set(xscale='log', xlabel='tau in ms', ylabel='Correlation G(tau)')
          ax5.set(xscale='log', xlabel='tau in ms', ylabel='Correlation G(tau)')
          ax6 = fig.add_subplot(gs[i*4+2:i*4+4, 2], title='multipletau w/ us bin', sharey=ax4)
          sns.lineplot(ax=ax6, data=cdm_v, marker='.', legend=False)
          sns.lineplot(ax=ax6, data=fdm_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax7 = fig.add_subplot(gs[i*4+2:i*4+4, 3], title='tttr2xfcs on arrival times', sharey=ax5)
          sns.lineplot(ax=ax7, data=cdt_v, marker='.', legend=False)
          sns.lineplot(ax=ax7, data=fdt_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax6.set(xscale='log', xlabel='tau in ms', ylabel='Correlation G(tau)')
          ax7.set(xscale='log', xlabel='tau in ms', ylabel='Correlation G(tau)')
      fig.suptitle('HsPEX5-eGFP (left) vs TbPEX5-eGFP (right) - 11.11.2019', size=20)
      gs.tight_layout(fig, rect=[0, 0, 1, 0.99])
      
      
    • the results: analysis4-191111-pex1.png
    • cave 1: 2 channels - check for the right Channel when plotting (CH2)
    • cave 2: in the plot, the traces and the correlations don't fit right now inside their respecitve group (different order of names in e.g. traces_clean vs fit_clean_mt) → at the moment I don't invest the time to fix this, because the fits are quite similar looking, but for publication I have to do this.
  • look at third bright cluster artifacts folder: 191113_Pex5_2_structured
    • clean = Homo sapiens PEX5, labelled with eGFP
    • dirty = Trypanosoma brucei PEX5, labelled with eGFP
    • let's plot traces, predictions, and correlations (see Details block):
      path_clean = data_path / '191113_Pex5_2_structured/HsPEX5EGFP_1-100001_3of250'
      path_dirty = data_path / '191113_Pex5_2_structured/TbPEX5EGFP_1-10002_3of250'
      files_clean = [path_clean / f for f in os.listdir(path_clean) if f.endswith('.ptu')]
      files_dirty = [path_dirty / f for f in os.listdir(path_dirty) if f.endswith('.ptu')]
      traces = pd.DataFrame()
      predtraces = pd.DataFrame()
      preds = pd.DataFrame()
      corrs = pd.DataFrame()
      mt_bin = 1e-3
      
      
      for myfile in (files_clean):
          ptufile = cfo.PicoObject(myfile, par_obj)
          # ptufile.predictTimeSeries(model=loaded_model,
          #                           scaler='minmax')
          mt_name = f'us_{ptufile.name}'
          ptufile.getTimeSeries(timeseries_name=mt_name,
                                photonCountBin=mt_bin)
          ptufile.getPhotonCountingStats(name=mt_name)
          # ptufile.predictTimeSeries(model=loaded_model,
          #                           scaler='minmax',
          #                           name=mt_name)
          for key in list(ptufile.trueTimeArr.keys()):
              ptufile.get_autocorrelation(method='tttr2xfcs', name=key)
          for key in list(ptufile.timeSeries.keys()):
              for k, i in ptufile.timeSeries[key].items():
                  if "PREPRO" in k:
                      if "1.0" in k:
                          # predtraces = pd.concat([predtraces, pd.DataFrame(i, columns=[f'{key}_{k}'])],
                          #                        axis=1)
                          pass
                  elif "1.0" in k:
                      traces = pd.concat([traces, pd.DataFrame(i, columns=[f'{key}_{k}'])],
                                         axis=1)
                      # preds = pd.concat([preds, pd.DataFrame(data=ptufile.predictions[key][k],
                      #                                        columns=[f'{key}_{k}'])], axis=1)
                  elif "0.001" in k:
                      ptufile.get_autocorrelation(method='multipletau', name=(key, k))
      
          for m in ['multipletau', 'tttr2xfcs', 'tttr2xfcs_with_weights']:
              if m in list(ptufile.autoNorm.keys()):
                  for key, item in list(ptufile.autoNorm[m].items()):
                      ptufile.save_autocorrelation(name=key, method=m,
                                                   output_path=output_path)
      # predtraces.to_csv(Path(output_path) / 'pex2dirty_predtraces.csv')
      traces.to_csv(Path(output_path) / 'pex2clean_traces.csv')
      # preds.to_csv(Path(output_path) / 'pex2dirty_preds.csv')
      
      /home/lex/Programme/miniconda3/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      
      fit_dirty_mt = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/dirty/pex2dirty_multipletau_plot.csv',
                                     index_col=0, usecols=[0, 2, 4, 6], na_values=' ')
      fit_dirty_tt = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/dirty/pex2dirty_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 2, 4, 6], na_values=' ')
      fit_clean_mt = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/clean/pex2clean_multipletau_plot.csv'
                                     , index_col=0, usecols=[0, 2, 4, 6], na_values=' ')
      fit_clean_tt = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/clean/pex2clean_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 2, 4, 6], na_values=' ')
      corr_dirty_mt = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/dirty/pex2dirty_multipletau_plot.csv',
                                     index_col=0, usecols=[0, 1, 3, 5], na_values=' ')
      corr_dirty_tt = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/dirty/pex2dirty_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 1, 3, 5], na_values=' ')
      corr_clean_mt = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/clean/pex2clean_multipletau_plot.csv'
                                     , index_col=0, usecols=[0, 1, 3, 5], na_values=' ')
      corr_clean_tt = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/clean/pex2clean_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 1, 3, 5], na_values=' ')
      
      # preds_dirty = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/dirty/pex2dirty_preds.csv', index_col=0)
      # predtraces_dirty = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/dirty/pex2dirty_predtraces.csv', index_col=0)
      traces_dirty = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/dirty/pex2dirty_traces.csv', index_col=0)
      # preds_clean = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/clean/pex2clean_preds.csv', index_col=0)
      # predtraces_clean = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/clean/pex2clean_predtraces.csv', index_col=0)
      traces_clean = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/clean/pex2clean_traces.csv', index_col=0)
      fit_clean_mt.columns
      
      Index(['2022-02-02_multipletau_CH2_BIN0dot001_us_HsPEX5EGFP 1-1000015_T309s_1_correlation-CH2_2 fitted model: ',
             '2022-02-02_multipletau_CH2_BIN0dot001_us_HsPEX5EGFP 1-1000016_T331s_1_correlation-CH2_2 fitted model: ',
             '2022-02-02_multipletau_CH2_BIN0dot001_us_HsPEX5EGFP 1-1000017_T353s_1_correlation-CH2_2 fitted model: '],
            dtype='object')
      
      plotdata = zip(traces_dirty.items(), traces_clean.items(),
                     corr_dirty_mt.items(), corr_clean_mt.items(), corr_dirty_tt.items(),
                     corr_clean_tt.items(), fit_dirty_mt.items(), fit_clean_mt.items(),
                     fit_dirty_tt.items(), fit_clean_tt.items())
      fig = plt.figure(figsize=(16, 20))
      gs = fig.add_gridspec(12, 4)
      for i, ((_, td_v), (_, tc_v), (_, cdm_v), (_, ccm_v), (_, cdt_v), (_, cct_v),
              (_, fdm_v), (_, fcm_v), (_, fdt_v), (_, fct_v)) in enumerate(plotdata):
          print(i)
          ax0 = fig.add_subplot(gs[i*4, 0:2])
          sns.lineplot(ax=ax0, data=tc_v)
          ax1 = fig.add_subplot(gs[i*4, 2:4])
          sns.lineplot(ax=ax1, data=td_v)
          ax0.set(xlabel='time steps in ms', ylabel='photons')
          ax1.set(xlabel='time steps in ms', ylabel='photons')
          # ax2 = fig.add_subplot(gs[i*4+1, 0:2], sharex=ax0)
          # sns.lineplot(ax=ax2, data=ptc_v, color=sns.color_palette()[0], legend=False)
          # sns.lineplot(ax=ax2, data=pc_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          # ax3 = fig.add_subplot(gs[i*4+1, 2:4], sharex=ax1)
          # sns.lineplot(ax=ax3, data=ptd_v, legend=False)
          # sns.lineplot(ax=ax3, data=pd_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          # ax2.set(xlabel='time steps in ms', ylabel='probability of artifact')
          # ax3.set(xlabel='time steps in ms', ylabel='probability of artifact')
          ax4 = fig.add_subplot(gs[i*4+2:i*4+4, 0], title='multipletau w/ us bin')
          sns.lineplot(ax=ax4, data=ccm_v, marker='.', legend=False)
          sns.lineplot(ax=ax4, data=fcm_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax5 = fig.add_subplot(gs[i*4+2:i*4+4, 1], title='tttr2xfcs on arrival times')
          sns.lineplot(ax=ax5, data=cct_v, marker='.', legend=False)
          sns.lineplot(ax=ax5, data=fct_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax4.set(xscale='log', xlabel='tau in ms', ylabel='Correlation G(tau)')
          ax5.set(xscale='log', xlabel='tau in ms', ylabel='Correlation G(tau)')
          ax6 = fig.add_subplot(gs[i*4+2:i*4+4, 2], title='multipletau w/ us bin', sharey=ax4)
          sns.lineplot(ax=ax6, data=cdm_v, marker='.', legend=False)
          sns.lineplot(ax=ax6, data=fdm_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax7 = fig.add_subplot(gs[i*4+2:i*4+4, 3], title='tttr2xfcs on arrival times', sharey=ax5)
          sns.lineplot(ax=ax7, data=cdt_v, marker='.', legend=False)
          sns.lineplot(ax=ax7, data=fdt_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax6.set(xscale='log', xlabel='tau in ms', ylabel='Correlation G(tau)')
          ax7.set(xscale='log', xlabel='tau in ms', ylabel='Correlation G(tau)')
      fig.suptitle('HsPEX5-eGFP (left) vs TbPEX5-eGFP (right) - 13.11.2019', size=20)
      gs.tight_layout(fig, rect=[0, 0, 1, 0.99])
      
      
    • the results: analysis4-191113_pex2.png
  • look at detector dropout files
    • from the file Name we have e24 ? HeLa cells with GFP-SNAP
    • let's see if we can load them (see Details block)
      path_dirty = data_path / "190327_detectordropout/"
      files_dirty = [path_dirty / f for f in os.listdir(path_dirty) if f.endswith('.ptu')]
      traces = pd.DataFrame()
      predtraces = pd.DataFrame()
      preds = pd.DataFrame()
      corrs = pd.DataFrame()
      mt_bin = 1e-3
      
      for myfile in (files_dirty):
          ptufile = cfo.PicoObject(myfile, par_obj)
          # ptufile.predictTimeSeries(model=loaded_model,
          #                           scaler='minmax')
         #  mt_name = f'us_{ptufile.name}'
         #  ptufile.getTimeSeries(timeseries_name=mt_name,
         #                        photonCountBin=mt_bin)
         #  ptufile.getPhotonCountingStats(name=mt_name)
          # ptufile.predictTimeSeries(model=loaded_model,
          #                           scaler='minmax',
          #                           name=mt_name)
          for key in list(ptufile.trueTimeArr.keys()):
              ptufile.get_autocorrelation(method='tttr2xfcs', name=key)
          for key in list(ptufile.timeSeries.keys()):
              for k, i in ptufile.timeSeries[key].items():
                  if "PREPRO" in k:
                      if "1.0" in k:
                          # predtraces = pd.concat([predtraces, pd.DataFrame(i, columns=[f'{key}_{k}'])],
                          #                        axis=1)
                          pass
                  elif "1.0" in k:
                      traces = pd.concat([traces, pd.DataFrame(i, columns=[f'{key}_{k}'])],
                                         axis=1)
                      ptufile.get_autocorrelation(method='multipletau', name=(key, k))
                      # preds = pd.concat([preds, pd.DataFrame(data=ptufile.predictions[key][k],
                      #                                        columns=[f'{key}_{k}'])], axis=1)
                  elif "0.001" in k:
                      ptufile.get_autocorrelation(method='multipletau', name=(key, k))
      
          for m in ['multipletau', 'tttr2xfcs', 'tttr2xfcs_with_weights']:
              if m in list(ptufile.autoNorm.keys()):
                  for key, item in list(ptufile.autoNorm[m].items()):
                      ptufile.save_autocorrelation(name=key, method=m,
                                                   output_path=output_path)
      # predtraces.to_csv(Path(output_path) / 'detectordropout_predtraces.csv')
      traces.to_csv(Path(output_path) / 'detectordropout_traces.csv')
      # preds.to_csv(Path(output_path) / 'detectordropout_preds.csv')
      
      /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      
      fit_dirty = pd.read_csv(Path(output_path) / '190327_detectordropout_onlydirty/detectordropout_tttr2xfcs_plot.csv',
                              index_col=0, usecols=[0, 2, 4, 6, 8, 10], na_values=' ')
      corr_dirty = pd.read_csv(Path(output_path) / '190327_detectordropout_onlydirty/detectordropout_tttr2xfcs_plot.csv',
                               index_col=0, usecols=[0, 1, 3, 5, 7, 9], na_values=' ')
      
      # preds_dirty = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/dirty/pex2dirty_preds.csv', index_col=0)
      # predtraces_dirty = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/dirty/pex2dirty_predtraces.csv', index_col=0)
      traces_dirty = pd.read_csv(Path(output_path) / '190327_detectordropout_onlydirty/detectordropout_traces.csv', index_col=0)
      # preds_clean = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/clean/pex2clean_preds.csv', index_col=0)
      # predtraces_clean = pd.read_csv(Path(output_path) / '191113_Pex5_2_structured/clean/pex2clean_predtraces.csv', index_col=0)
      
      
      plotdata = zip(traces_dirty.items(), corr_dirty.items(), fit_dirty.items())
      fig = plt.figure(figsize=(16, 20))
      gs = fig.add_gridspec(9, 4)
      for i, ((td_n, td_v), (_, cdt), (_, fdt)) in enumerate(plotdata):
          print(i)
          if i in [0, 2, 4]:
              i = int(i*1.5)
              ax0 = fig.add_subplot(gs[i, 0:2], title=td_n)
              ax1 = fig.add_subplot(gs[i+1:i+3, 0], title='tttr2xfcs on arrival times')
          elif i in [1, 3]:
              i -= 1
              i = int(i*1.5)
              ax0 = fig.add_subplot(gs[i, 2:4], title=td_n)
              ax1 = fig.add_subplot(gs[i+1:i+3, 2], title='tttr2xfcs on arrival times')
          sns.lineplot(ax=ax0, data=td_v)
          ax0.set(xlabel='time steps in ms', ylabel='photons')
          sns.lineplot(ax=ax1, x=cdt.index, y=cdt, marker='.', legend=False)
          sns.lineplot(ax=ax1, x=fdt.index, y=fdt, color=sns.color_palette()[1], alpha=0.8, legend=False)
          xmin = fdt.dropna().index[0]
          xmax = fdt.dropna().index[-1]
          ymin = np.min(fdt.dropna())
          ymax = np.max(fdt.dropna())
          plt.setp(ax1, xlim=[xmin - 0.5*xmin, xmax + 0.5*xmax], ylim=[ymin - 0.5*ymin, ymax + 0.5 * ymax],
                   xscale='log', xlabel=r'$\tau$ (ms)', ylabel=r'Correlation G($\tau$)')
      fig.suptitle('E24-HeLa-GFP-SNAP - detectordropout - 13.11.2019', size=20)
      gs.tight_layout(fig, rect=[0, 0, 1, 0.99])
      
      
    • results: analysis4-190327-detectordropout.png
    • since each measurement is around 1 GB in size (and loading and processing this file as seen above needs around 100GB of RAM), the experimentors took more traces and saved out only the correlations.
    • in total we have 11 experiments with 3 repetitions each
      • so in chopIn01 we have 33 traces with a length of about 16 seconds
      • and in chopIn03 each trace is cut in 3 parts so we have 99 traces with a length of around 5 seconds
    • Let's load and plot them (see Details block):
      chop01_cf = pd.read_csv(Path(output_path) / '190327_detectordropout_onlydirty/E24 HeLa GFP_chopIn01_FCS_CurvesAndFits.csv',
                                      header=[0, 1])
      chop01_ts = pd.read_csv(Path(output_path) / '190327_detectordropout_onlydirty/E24 HeLa GFP_chopIn01_timetraces.csv',
                                      header=[0, 1])
      
      corr_tau_ind = [i for i in chop01_cf.columns if 'Correlation' in i[0]]
      corr_ind = [i for i in chop01_cf.columns if 'Correlation' in i[1]]
      corr_sd_ind = [i for i in chop01_cf.columns if 'Deviation' in i[0]]
      fit_ind = [i for i in chop01_cf.columns if 'Fit' in i[1]]
      fit_tau_ind = [i for i in chop01_cf.columns if 'Fit' in i[0]]
      res_tau_ind = [i for i in chop01_cf.columns if 'Residuals' in i[0]]
      res_ind = [i for i in chop01_cf.columns if 'Residuals' in i[1]]
      trace_tau_ind = [i for i in chop01_ts.columns if 'Count Rate' in i[0]]
      trace_ind = [i for i in chop01_ts.columns if 'Count Rate' in i[1]]
      
      fig = plt.figure(figsize=(16, 66))
      gs = fig.add_gridspec(33, 4)
      for i, _ in enumerate(corr_tau_ind):
          ct = chop01_cf[corr_tau_ind].iloc[:, i]
          corr = chop01_cf[corr_ind].iloc[:, i]
          csd = chop01_cf[corr_sd_ind].iloc[:, i]
          fit = chop01_cf[fit_ind].iloc[:, i]
          ft = chop01_cf[fit_tau_ind].iloc[:, i]
          res = chop01_cf[res_ind].iloc[:, i]
          rt = chop01_cf[res_tau_ind].iloc[:, i]
          trace = chop01_ts[trace_ind].iloc[:, i]
          tt = chop01_ts[trace_tau_ind].iloc[:, i]
      
          ax0 = fig.add_subplot(gs[i, 0:2], title=ct.name)
          ax1 = fig.add_subplot(gs[i, 2])
          ax2 = fig.add_subplot(gs[i, 3])
      
          sns.lineplot(ax=ax0, x=tt, y=trace)
          sns.lineplot(ax=ax1, x=ct, y=corr, marker='.', label='Correlation')
          sns.lineplot(ax=ax1, x=ft, y=fit, color=sns.color_palette()[1], alpha=0.8, label='Fit')
          sns.lineplot(ax=ax2, x=rt, y=res)
      
          plt.setp(ax0, ylabel='[kCounts/s]', xlabel='Time [s]')
          plt.setp(ax1, ylabel=r'Correlation G($\tau$)', xlabel=r'$\tau$ [ms]', xscale='log')
          plt.setp(ax2, ylabel='Residuals', xlabel=r'$\tau$ [ms]', xscale='log')
      fig.suptitle('E24-HeLa-GFP-SNAP - detector dropout - 11 traces, 3 repetitions, 1 part - 13.11.2019',
                   size=20, y=1.005)
      plt.tight_layout()
      
      
      • these are examples of traces cut in 3 parts: analysis4-190327-chopin01.png
      chop03_cf = pd.read_csv(Path(output_path) / '190327_detectordropout_onlydirty/E24 HeLa GFP_chopIn03_FCS_CurvesAndFits.csv',
                                      header=[0, 1])
      chop03_ts = pd.read_csv(Path(output_path) / '190327_detectordropout_onlydirty/E24 HeLa GFP_chopIn03_timetraces.csv',
                                      header=[0, 1])
      
      corr_tau_ind = [i for i in chop03_cf.columns if 'Correlation' in i[0]]
      corr_ind = [i for i in chop03_cf.columns if 'Correlation' in i[1]]
      corr_sd_ind = [i for i in chop03_cf.columns if 'Deviation' in i[0]]
      fit_ind = [i for i in chop03_cf.columns if 'Fit' in i[1]]
      fit_tau_ind = [i for i in chop03_cf.columns if 'Fit' in i[0]]
      res_tau_ind = [i for i in chop03_cf.columns if 'Residuals' in i[0]]
      res_ind = [i for i in chop03_cf.columns if 'Residuals' in i[1]]
      trace_tau_ind = [i for i in chop03_ts.columns if 'Count Rate' in i[0]]
      trace_ind = [i for i in chop03_ts.columns if 'Count Rate' in i[1]]
      
      fig = plt.figure(figsize=(16, 24))
      gs = fig.add_gridspec(8, 4)
      i = 0
      for j, _ in enumerate(corr_tau_ind):
          if j in [9, 10, 18, 19, 20, 46, 47, 48]:
              ct = chop03_cf[corr_tau_ind].iloc[:, j]
              corr = chop03_cf[corr_ind].iloc[:, j]
              csd = chop03_cf[corr_sd_ind].iloc[:, j]
              fit = chop03_cf[fit_ind].iloc[:, j]
              ft = chop03_cf[fit_tau_ind].iloc[:, j]
              res = chop03_cf[res_ind].iloc[:, j]
              rt = chop03_cf[res_tau_ind].iloc[:, j]
              trace = chop03_ts[trace_ind].iloc[:, j]
              tt = chop03_ts[trace_tau_ind].iloc[:, j]
      
              ax0 = fig.add_subplot(gs[i, 0:2], title=ct.name)
              ax1 = fig.add_subplot(gs[i, 2])
              ax2 = fig.add_subplot(gs[i, 3])
      
              sns.lineplot(ax=ax0, x=tt, y=trace)
              sns.lineplot(ax=ax1, x=ct, y=corr, marker='.', label='Correlation')
              sns.lineplot(ax=ax1, x=ft, y=fit, color=sns.color_palette()[1], alpha=0.8, label='Fit')
              sns.lineplot(ax=ax2, x=rt, y=res)
      
              plt.setp(ax0, ylabel='[kCounts/s]', xlabel='Time [s]')
              plt.setp(ax1, ylabel=r'Correlation G($\tau$)', xlabel=r'$\tau$ [ms]', xscale='log')
              plt.setp(ax2, ylabel='Residuals', xlabel=r'$\tau$ [ms]', xscale='log')
              i += 1
      fig.suptitle('E24-HeLa-GFP-SNAP - detector dropout - 11 traces, 3 repetitions, 3 parts (example) - 13.11.2019',
                   size=20, y=1.005)
      plt.tight_layout()
      
      
      • these are examples of traces cut in 9 parts: analysis4-190327-chopin03.png
    • now let's load and plot the transit times (see Details block):
      chop01 = pd.read_csv(Path(output_path) / '190327_detectordropout_onlydirty/E24 HeLa GFP_chopIn01_FitParameters.csv',
                                header=[0], index_col=1)
      chop03 = pd.read_csv(Path(output_path) / '190327_detectordropout_onlydirty/E24 HeLa GFP_chopIn03_FitParameters.csv',
                                header=[0], index_col=1)
      chop10 = pd.read_csv(Path(output_path) / '190327_detectordropout_onlydirty/E24 HeLa GFP_chopIn10_FitParameters.csv',
                                header=[0], index_col=1)
      taudiff01 = chop01.loc[:, 'Diffusion Time 1 µs']
      taudiff03 = chop03.loc[:, 'Diffusion Time 1 µs']
      taudiff10 = chop10.loc[:, 'Diffusion Time 1 µs']
      taudiff_dirty01 = taudiff01.loc[[4, 7, 16, 17]].rename('dirty\n(16s, n=4)')
      taudiff_clean01 = taudiff01.drop([4, 7, 16, 17]).rename('clean\n(16s, n=29)')
      taudiff_dirty03 = taudiff03.loc[[11, 19, 21, 47, 49]].rename('dirty\n(5s, n=5)')
      taudiff_clean03 = taudiff03.drop([11, 19, 21, 47, 49]).rename('clean\n(5s, n=94)')
      taudiff_dirty10 = taudiff10.loc[[34, 62, 67, 157]].rename('dirty\n(1.6s, n=4)')
      taudiff_clean10 = taudiff10.drop([34, 62, 67, 157]).rename('clean\n(1.6s, n=326)')
      taudiff_plot = pd.concat([taudiff_clean01, taudiff_clean03, taudiff_clean10,
                                taudiff_dirty01, taudiff_dirty03, taudiff_dirty10],
                               axis=1)
      
      ax = sns.boxplot(data=taudiff_plot)
      sns.stripplot(data=taudiff_plot, dodge=True, palette=sns.color_palette(['0.3']))
      plt.setp(ax, ylabel=r'transit time [$\mu s$]', yscale='log',
               title='E24-HeLa-GFP-SNAP - with (dirty) vs without (clean) detector dropout\n11 traces, 3 repetitions - 13.11.2019')
      ax.set_xticklabels(ax.get_xticklabels(), rotation=45)
      
      • the results: analysis4-190327-all-param.png
    • it would be possible to train an UNET on simulated data with such detector dropout artifacts and test it on the above 4 traces with artifacts I have as .ptu files. Unfortunately, I only have 1 "clean" .ptu file as a negative control.
  • look at photobleaching files
    • we have eGFP 0.5 ug/ml in solution
      • clean = laser power 5 uW, 5 traces
      • dirty = laser power 10 uW,
    • let's see if we can load them
      • look at detector dropout files
    • from the file Name we have e24 ? HeLa cells with GFP-SNAP
    • let's see if we can load them (see Details block)
      path_clean = data_path / "191107_EGFP_photobleaching/eGFP 5 uW 0.5ugml1"
      path_dirty = data_path / "191107_EGFP_photobleaching/eGFP 10 uW 0.5ugml2"
      files_clean = [path_clean / f for f in os.listdir(path_clean) if f.endswith('.ptu')]
      files_dirty = [path_dirty / f for f in os.listdir(path_dirty) if f.endswith('.ptu')]
      traces = pd.DataFrame()
      predtraces = pd.DataFrame()
      preds = pd.DataFrame()
      corrs = pd.DataFrame()
      mt_bin = 1e-3
      
      for myfile in (files_dirty):
          ptufile = cfo.PicoObject(myfile, par_obj)
          # ptufile.predictTimeSeries(model=loaded_model,
          #                           scaler='minmax')
          mt_name = f'us_{ptufile.name}'
          ptufile.getTimeSeries(timeseries_name=mt_name,
                                photonCountBin=mt_bin)
          ptufile.getPhotonCountingStats(name=mt_name)
          # ptufile.predictTimeSeries(model=loaded_model,
          #                           scaler='minmax',
          #                           name=mt_name)
          for key in list(ptufile.trueTimeArr.keys()):
              ptufile.get_autocorrelation(method='tttr2xfcs', name=key)
          for key in list(ptufile.timeSeries.keys()):
              for k, i in ptufile.timeSeries[key].items():
                  if "PREPRO" in k:
                      if "1.0" in k:
                          # predtraces = pd.concat([predtraces, pd.DataFrame(i, columns=[f'{key}_{k}'])],
                          #                        axis=1)
                          pass
                  elif "1.0" in k:
                      traces = pd.concat([traces, pd.DataFrame(i, columns=[f'{key}_{k}'])],
                                         axis=1)
                      # preds = pd.concat([preds, pd.DataFrame(data=ptufile.predictions[key][k],
                      #                                        columns=[f'{key}_{k}'])], axis=1)
                  elif "0.001" in k:
                      ptufile.get_autocorrelation(method='multipletau', name=(key, k))
      
          for m in ['multipletau', 'tttr2xfcs', 'tttr2xfcs_with_weights']:
              if m in list(ptufile.autoNorm.keys()):
                  for key, item in list(ptufile.autoNorm[m].items()):
                      ptufile.save_autocorrelation(name=key, method=m,
                                                   output_path=output_path)
      # predtraces.to_csv(Path(output_path) / 'detectordropout_predtraces.csv')
      traces.to_csv(Path(output_path) / 'photobleaching_traces_dirty.csv')
      # preds.to_csv(Path(output_path) / 'detectordropout_preds.csv')
      
      /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/multipletau/core.py:175: DtypeWarning: Input dtype is not float; casting to np.float_!
        warnings.warn("Input dtype is not float; casting to np.float_!",
      
      fit_dirty_mt = pd.read_csv(Path(output_path) / '191107_photobleaching/dirty/10uW_CH2_multipletau_plot.csv',
                                     index_col=0, usecols=[0, 2, 4, 6, 8, 10], na_values=' ')
      fit_dirty_tt = pd.read_csv(Path(output_path) / '191107_photobleaching/dirty/10uW_CH2_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 2, 4, 6, 8, 10], na_values=' ')
      fit_clean_mt = pd.read_csv(Path(output_path) / '191107_photobleaching/clean/5uW_CH2_multipletau_plot.csv'
                                     , index_col=0, usecols=[0, 2, 4, 6, 8, 10], na_values=' ')
      fit_clean_tt = pd.read_csv(Path(output_path) / '191107_photobleaching/clean/5uW_CH2_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 2, 4, 6, 8, 10], na_values=' ')
      corr_dirty_mt = pd.read_csv(Path(output_path) / '191107_photobleaching/dirty/10uW_CH2_multipletau_plot.csv',
                                     index_col=0, usecols=[0, 1, 3, 5, 7, 9], na_values=' ')
      corr_dirty_tt = pd.read_csv(Path(output_path) / '191107_photobleaching/dirty/10uW_CH2_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 1, 3, 5, 7, 9], na_values=' ')
      corr_clean_mt = pd.read_csv(Path(output_path) / '191107_photobleaching/clean/5uW_CH2_multipletau_plot.csv'
                                     , index_col=0, usecols=[0, 1, 3, 5, 7, 9], na_values=' ')
      corr_clean_tt = pd.read_csv(Path(output_path) / '191107_photobleaching/clean/5uW_CH2_tttr2xfcs_plot.csv'
                                     , index_col=0, usecols=[0, 1, 3, 5, 7, 9], na_values=' ')
      
      # preds_dirty = pd.read_csv(Path(output_path) / '191107_photobleaching/dirty/pex1dirty_preds.csv', index_col=0)
      # predtraces_dirty = pd.read_csv(Path(output_path) / '191107_photobleaching/dirty/pex1dirty_predtraces.csv', index_col=0)
      traces_dirty = pd.read_csv(Path(output_path) / '191107_photobleaching/dirty/photobleaching_traces_dirty.csv',
                                 index_col=0, usecols=[0, 2, 4, 6, 8, 10])
      # preds_clean = pd.read_csv(Path(output_path) / '191107_photobleaching/clean/pex1clean_preds.csv', index_col=0)
      # predtraces_clean = pd.read_csv(Path(output_path) / '191107_photobleaching/clean/pex1clean_predtraces.csv', index_col=0)
      traces_clean = pd.read_csv(Path(output_path) / '191107_photobleaching/clean/photobleaching_traces_clean.csv',
                                 index_col=0, usecols=[0, 2, 4, 6, 8, 10])
      print(fit_clean_mt.columns)
      print(traces_clean.columns)
      
      Index(['2022-02-19_multipletau_CH2_BIN0dot001_us_eGFP1-5uW_T0s_1_correlation-CH2_2 fitted model: ',
             '2022-02-19_multipletau_CH2_BIN0dot001_us_eGFP2-5uW_T25s_1_correlation-CH2_2 fitted model: ',
             '2022-02-19_multipletau_CH2_BIN0dot001_us_eGFP3-5uW_T48s_1_correlation-CH2_2 fitted model: ',
             '2022-02-19_multipletau_CH2_BIN0dot001_us_eGFP4-5uW_T69s_1_correlation-CH2_2 fitted model: ',
             '2022-02-19_multipletau_CH2_BIN0dot001_us_eGFP5-5uW_T91s_1_correlation-CH2_2 fitted model: '],
            dtype='object')
      Index(['eGFP1-5uW_T0s_1_CH2_BIN1.0', 'eGFP4-5uW_T69s_1_CH2_BIN1.0',
             'eGFP2-5uW_T25s_1_CH2_BIN1.0', 'eGFP3-5uW_T48s_1_CH2_BIN1.0',
             'eGFP5-5uW_T91s_1_CH2_BIN1.0'],
            dtype='object')
      
      plotdata = zip(traces_dirty.items(), traces_clean.items(),
                     corr_dirty_mt.items(), corr_clean_mt.items(), corr_dirty_tt.items(),
                     corr_clean_tt.items(), fit_dirty_mt.items(), fit_clean_mt.items(),
                     fit_dirty_tt.items(), fit_clean_tt.items())
      fig = plt.figure(figsize=(16, 30))
      gs = fig.add_gridspec(15, 4)
      for i, ((td_n, td_v), (tc_n, tc_v), (_, cdm_v), (_, ccm_v), (_, cdt_v), (_, cct_v),
              (_, fdm_v), (_, fcm_v), (_, fdt_v), (_, fct_v)) in enumerate(plotdata):
          print(i)
          ax0 = fig.add_subplot(gs[i*3, 0:2], title=tc_n)
          sns.lineplot(ax=ax0, data=tc_v, legend='auto')
          ax1 = fig.add_subplot(gs[i*3, 2:4], title=td_n)
          sns.lineplot(ax=ax1, data=td_v, legend='auto')
          ax0.set(xlabel='time steps in ms', ylabel='photons')
          ax1.set(xlabel='time steps in ms', ylabel='photons')
          # ax2 = fig.add_subplot(gs[i*4+1, 0:2], sharex=ax0)
          # sns.lineplot(ax=ax2, data=ptc_v, color=sns.color_palette()[0], legend=False)
          # sns.lineplot(ax=ax2, data=pc_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          # ax3 = fig.add_subplot(gs[i*4+1, 2:4], sharex=ax1)
          # sns.lineplot(ax=ax3, data=ptd_v, legend=False)
          # sns.lineplot(ax=ax3, data=pd_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          # ax2.set(xlabel='time steps in ms', ylabel='probability of artifact')
          # ax3.set(xlabel='time steps in ms', ylabel='probability of artifact')
          ax4 = fig.add_subplot(gs[i*3+1:i*3+3, 0], title=r'multipletau w/ $\mu s$ bin')
          sns.lineplot(ax=ax4, data=ccm_v, marker='.', legend=False)
          sns.lineplot(ax=ax4, data=fcm_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax5 = fig.add_subplot(gs[i*3+1:i*3+3, 1], title='tttr2xfcs on arrival times')
          sns.lineplot(ax=ax5, data=cct_v, marker='.', legend=False)
          sns.lineplot(ax=ax5, data=fct_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax4.set(xscale='log', xlabel=r'$\tau$ [ms]', ylabel=r'Correlation G($\tau$)')
          ax5.set(xscale='log', xlabel=r'$\tau$ [ms]', ylabel=r'Correlation G($\tau$)')
          ax6 = fig.add_subplot(gs[i*3+1:i*3+3, 2], title=r'multipletau w/ $\mu s$ bin', sharey=ax4)
          sns.lineplot(ax=ax6, data=cdm_v, marker='.', legend=False)
          sns.lineplot(ax=ax6, data=fdm_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax7 = fig.add_subplot(gs[i*3+1:i*3+3, 3], title='tttr2xfcs on arrival times', sharey=ax5)
          sns.lineplot(ax=ax7, data=cdt_v, marker='.', legend=False)
          sns.lineplot(ax=ax7, data=fdt_v, color=sns.color_palette()[1], alpha=0.8, legend=False)
          ax6.set(xscale='log', xlabel=r'$\tau$ [ms]', ylabel=r'Correlation G($\tau$)')
          ax7.set(xscale='log', xlabel=r'$\tau$ [ms]', ylabel=r'Correlation G($\tau$)')
      fig.suptitle('eGFP and 5uW (left, clean) vs eGFP and 10uW (right, dirty) - 07.11.2019', size=20)
      gs.tight_layout(fig, rect=[0, 0, 1, 0.99])
      
      
    • results:

    analysis4-191107-photobleaching.png

    • seems like this this data does not contain so much of an artifact. Have to ask Pablo if there was other data he took.
    • let's see if the transit times are different (see Details block):
      taudiff_dirty_mt = pd.read_csv(Path(output_path) / '191107_photobleaching/dirty/10uW_CH2_multipletau_param.csv',
                                     na_values=' ').loc[:, 'txy1'].rename('dirty (n=5\nmultipletau)')
      taudiff_dirty_tt = pd.read_csv(Path(output_path) / '191107_photobleaching/dirty/10uW_CH2_tttr2xfcs_param.csv',
                                     na_values=' ').loc[:, 'txy1'].rename('dirty (n=5\ntttr2xfcs)')
      taudiff_clean_mt = pd.read_csv(Path(output_path) / '191107_photobleaching/clean/5uW_CH2_multipletau_param.csv',
                                     na_values=' ').loc[:, 'txy1'].rename('clean (n=5\nmultipletau)')
      taudiff_clean_tt = pd.read_csv(Path(output_path) / '191107_photobleaching/clean/5uW_CH2_tttr2xfcs_param.csv',
                                     na_values=' ').loc[:, 'txy1'].rename('clean (n=5\ntttr2xfcs)')
      
      taudiff_plot = pd.concat([taudiff_clean_mt, taudiff_clean_tt,
                                taudiff_dirty_mt, taudiff_dirty_tt], axis=1)
      
      ax = sns.boxplot(data=taudiff_plot)
      sns.stripplot(data=taudiff_plot, dodge=True, palette=sns.color_palette(['0.3']))
      plt.setp(ax, ylabel=r'transit time [$\mu s$]', yscale='log',
               title='eGFP - 10uW (dirty) vs 5uW (clean) photobleaching - 07.11.2019')
      ax.set_xticklabels(ax.get_xticklabels(), rotation=45)
      
      
    • results: analysis4-191107-all-param.png
    • here too we don't see big differences in transit times.

2.5.12 learnings from exp-220120-correlate-ptu

  • in terms of biological experiments:
    • always double-check your data - in the process I mixed up the Alexa-LUV-experiment and the pex5 experiment - which made finding explanations harder when the correction didn't work out
    • it seems like I don't have photobleaching data yet → have to ask Pablo if there are other experimental data he took
  • in terms of machine learning and neural networks:
    • while plotting the alexa-luv data, I noticed that my network still has a bug - it learned on data with a fixed size of 2**14. When I use longer traces, I have to pad the data - here with the median value of the trace. BUT: the network wasn't trained on data like this. In this case, this leads to more false positives (predicted artifactual, even though clean). Re-training the model with such data helps.
    • while plotting the pex5 data, I noticed that prediction doesn't work with trace lengths of 2**15 → this can also be handled by re-training the network and adjusting the parameters so that it works with different input sizes (this works because it is a fully convolutional model)
    • from exp-210807-hparams I know that there are still more different network architectures that worked → I should train them as well, just to show that a wide range of architectures can solve this problem.
  • in terms of FCS autocorrelation and fitting:
    • there is still an open question if 2 component fitting is a way to improve the outcome of the correction. Dominic did a test which yielded some promising data - BUT he used a 2D fitting algorithm, and if I tried to play around with 3D fitting as described above, 1 component fitted nicely, but the second component always stayed at txy=0.01, alpha=1, A=1
    • also regarding fitting, even though delete_and_shift seems to be a promising method to correct bright cluster-artifacts in FCS, there is a shoulder in the correlation curve which is not fitted - this is probably expected since the LUVs have an expected speed at that area (~2-3ms), but still it should be further investigated
    • regarding the weights correction → it is not clear yet, why it doesn't work. The random correlation results show, that the bad results are probably not the result of an artifactual correlation of the weighted photons.
      • a possible explanation is, that with weight=0 or other low weights the gaps themselves get correlated → this happens also in the "detector dropout" artifact.
      • a possible follow-up experiment would be to distribute the photons in an artifactual bin randomly to remove any correlation in them.
    • regarding the weights=1-pred correction → it is also not clear, why it performs so badly.
    • regarding the delete and weights=0 corrections it is not clear, why they are not equal → have to re-do them in the same experiment.

2.5.13 Analysis 5: fitting with Focuspoint instead of focus-fit-js

  • Attention: this section was added after the branch exp-220120-correlate-ptu was merged into data.
  • it turns out, that there is a software error in focus-fit-js which makes 3D plots unreliable. Together with Pablo, we double-checked the data acquired with the standalone software FoCuS-point-correlator, written by Dominic as well, and tested for years.
  • With a 3D fit now working, we tried to do a 2 component fit on the data and see if we can regain the underlying transit times of the fast and slow components
  • now let's look at the results:
    %cd /home/lex/Programme/drmed-git
    
    /home/lex/Programme/drmed-git
    
    import numpy as np
    import matplotlib.pyplot as plt
    import pandas as pd
    import seaborn as sns
    from pathlib import Path
    from pprint import pprint
    # use seaborn style as default even if I just use matplotlib
    sns.set()
    sns.set_palette('colorblind')
    
  • load in all the data (see Details block)
    path = Path('data/exp-220120-correlate-ptu/2022-03-25_FoCuS-point-analysis')
    
    # clean params
    clean_nocorr_notrip_1spec_param_path = path / 'clean-nocorrection-notriplet-1species-5AR_outputParam.csv'
    # clean_nocorr_tripfixed_1spec_param_path = path / 'clean-nocorrection-triplet2a-0dot25-B1-0dot01-tauT1_outputParam.csv'
    
    # dirty params
    dirty_nocorr_notrip_2spec_param_path = path / 'dirty-nocorrection-notriplet-2species-5AR_outputParam.csv'
    # dirty_nocorr_tripsemifixed_2spec_param_path = path / 'dirty-nocorrection-triplet2a-float-B1-0dot01-tauT1-2species-5AR_outputParam.csv'
    # dirty_nocorr_tripfixed_2spec_param_path = path / 'dirty-nocorrection-triplet2a-0dot25-B1-0dot01-tauT1-2species-5AR_outputParam.csv'
    
    # dirty params
    dirty_cutshift_notrip_2spec_param_path = path / 'dirty-cutandshift-notriplet-2species-5AR_outputParam.csv'
    # dirty_cutshift_tripsemifixed_2spec_param_path = path / 'dirty-cutandshift-triplet2a-float-B1-0dot01-tauT1-2species-5AR_outputParam.csv'
    # dirty_cutshift_tripfixed_2spec_param_path = path / 'dirty-cutandshift-triplet2a-0dot25-B1-0dot01-tauT1-2species-5AR_outputParam.csv'
    dirty_delete_notrip_2spec_param_path = path / 'dirty-delete-notriplet-2species-5AR_outputParam.csv'
    dirty_weight0_notrip_2spec_param_path = path / 'dirty-weight0-notriplet-2species-5AR_outputParam.csv'
    dirty_1pred_notrip_2spec_param_path = path / 'dirty-1-pred-notriplet-2species-5AR_outputParam.csv'
    
    # clean params
    clean_param =  pd.read_csv(clean_nocorr_notrip_1spec_param_path, sep=',').assign(
        triplet=229*['None',], species=229*['1',], dimension=229*['3D eq 1B, AR=5',], fraction=229*['small',],
        correlator=229*['tttr2xfcs',], artifact=229*['clean',], correction=229*['No correction',])
    # clean_param =  pd.read_csv(clean_nocorr_tripfixed_1spec_param_path, sep=',').assign(
    #     triplet=229*['B1=0.25, tauT1=0.01',], species=229*['1',], dimension=229*['3D eq 1B, AR=5',],
    #     correlator=229*['tttr2xfcs',], artifact=229*['clean',], correction=229*['No correction',])
    # clean_param['txy2'] = clean_param['txy1']  # for plotting purposes
    # clean_param['A2'] = clean_param['A1']  # for plotting purposes
    
    # dirty params
    dirty_nocorr =  pd.read_csv(dirty_nocorr_notrip_2spec_param_path, sep=',').assign(
        triplet=173*['None',], species=173*['2',], dimension=173*['3D eq 1B, AR=5',], fraction=173*['small',],
        correlator=173*['tttr2xfcs',], artifact=173*['dirty',], correction=173*['No correction',])
    # dirty_nocorr_tripsemi =  pd.read_csv(dirty_nocorr_tripsemifixed_2spec_param_path, sep=',').assign(
    #     triplet=173*['B1=float, tauT1=0.01',], species=173*['2, AR=5',],
    #     correlator=173*['tttr2xfcs',], artifact=173*['dirty',], correction=173*['None',])
    # dirty_nocorr_tripfix =  pd.read_csv(dirty_nocorr_tripfixed_2spec_param_path, sep=',').assign(
    #     triplet=173*['B1=0.25, tauT1=0.01',], species=173*['2, AR=5',],
    #     correlator=173*['tttr2xfcs',], artifact=173*['dirty',], correction=173*['None',])
    dirty_weight0 =  pd.read_csv(dirty_weight0_notrip_2spec_param_path, sep=',').assign(
        triplet=173*['None',], species=173*['2',], dimension=173*['3D eq 1B, AR=5',], fraction=173*['small',],
        correlator=173*['tttr2xfcs',], artifact=173*['dirty',], correction=173*['weight=0',])
    
    
    dirty_cutshift =  pd.read_csv(dirty_cutshift_notrip_2spec_param_path, sep=',').assign(
        triplet=53*['None',], species=53*['2',], dimension=53*['3D eq 1B, AR=5',], fraction=53*['small',],
        correlator=53*['tttr2xfcs',], artifact=53*['dirty',], correction=53*['cut and shift',])
    # dirty_cutshift_tripsemi =  pd.read_csv(dirty_cutshift_tripsemifixed_2spec_param_path, sep=',').assign(
    #     triplet=53*['B1=float, tauT1=0.01',], species=53*['2, AR=5',],
    #     correlator=53*['tttr2xfcs',], artifact=53*['dirty',], correction=53*['cut and shift',])
    # dirty_cutshift_tripfix =  pd.read_csv(dirty_cutshift_tripfixed_2spec_param_path, sep=',').assign(
    #     triplet=53*['B1=0.25, tauT1=0.01',], species=53*['2, AR=5',],
    #     correlator=53*['tttr2xfcs',], artifact=53*['dirty',], correction=53*['cut and shift',])
    dirty_delete =  pd.read_csv(dirty_delete_notrip_2spec_param_path, sep=',').assign(
        triplet=53*['None',], species=53*['2',], dimension=53*['3D eq 1B, AR=5',], fraction=53*['small',],
        correlator=53*['tttr2xfcs',], artifact=53*['dirty',], correction=53*['delete',])
    dirty_1pred =  pd.read_csv(dirty_1pred_notrip_2spec_param_path, sep=',').assign(
        triplet=53*['None',], species=53*['2',], dimension=53*['3D eq 1B, AR=5',], fraction=53*['small',],
        correlator=53*['tttr2xfcs',], artifact=53*['dirty',], correction=53*['weight=1-prediction',])
    
    all_param = pd.concat([clean_param, dirty_nocorr, dirty_weight0,
                           dirty_cutshift, dirty_delete, dirty_1pred],
                          ignore_index=True)
    
    all_param["artifact-correction"] = all_param[["artifact", "correction"]].agg(' - '.join, axis=1)
    all_param = pd.wide_to_long(all_param, stubnames='txy', i=['name_of_plot'], j='fitted species (txy)')
    
    all_param = all_param.reset_index()
    all_param = pd.wide_to_long(all_param, stubnames='A', i=['name_of_plot', 'fitted species (txy)'], j='fitted species (A)')
    all_param = all_param.reset_index()
    
    
    g = sns.catplot(data=all_param,
                    y='txy',
                    x='artifact-correction',
                    hue='fitted species (txy)',
                    sharey=True,
                    height=5,
                    aspect=2,
                    legend_out=True,
                    kind='boxen',
                    showfliers=False)
    g.map_dataframe(sns.stripplot,
          y='txy',
          x='artifact-correction',
          hue='fitted species (txy)',
          dodge=True,
          palette=sns.color_palette(['0.3']),
          size=4,
          jitter=0.2)
    g.tight_layout()
    g.fig.suptitle('Transit times of AlexaFluor488 (clean) vs \nAlexaFluor488 + Dio LUVs (dirty) with different correction methods',
                   y=1.08, size=20)
    for axes in g.axes.flat:
         _ = axes.set_xticklabels(axes.get_xticklabels(), rotation=45)
    plt.setp(g.axes, yscale='log', xlabel='biological sample - correction method',
             ylabel=r'transit time $\tau_{D}$ (log)')
    plt.show()
    
    g = sns.catplot(data=all_param,
                    y='A',
                    x='artifact-correction',
                    hue='fitted species (A)',
                    sharey=True,
                    height=5,
                    aspect=2,
                    legend_out=True,
                    kind='boxen',
                    showfliers=False)
    g.map_dataframe(sns.stripplot,
          y='A',
          x='artifact-correction',
          hue='fitted species (A)',
          dodge=True,
          palette=sns.color_palette(['0.3']),
          size=4,
          jitter=0.2)
    g.tight_layout()
    g.fig.suptitle('Fraction sizes of AlexaFluor488 (clean) vs\nAlexaFluor488 + Dio LUVs (dirty) with different correction methods',
                   y=1.08, size=20)
    for axes in g.axes.flat:
         _ = axes.set_xticklabels(axes.get_xticklabels(), rotation=45)
    plt.setp(g.axes, xlabel='biological sample - correction method',
             ylabel='relative fraction size')
    plt.show()
    
  • let's have a look at transit times and fraction sizes of AF488 vs AF488 + DioLUVs + different correction methods analysis5-all-param-transit-times.png analysis5-all-param-fraction-sizes.png
  • I also had a look at different triplet parameters, we decided in the end against using them for the sake of simplicity: analysis5-2spec-triplet-3D-transit-times.png analysis5-2spec-triplet-3D-fraction-size.png
  • let's save the DataFrame used for plotting so that re-using it is simpler:
    all_param.to_csv('data/exp-220120-correlate-ptu/2022-04-05_all-params.csv')
    

2.6 exp-220227-unet

2.6.1 Setup: GPU node on HPC

  1. Setup tmux
    rm: cannot remove 'home/lex.tmux-local-socket-remote-machine': No such file or directory
    ye53nis@ara-login01.rz.uni-jena.de's password:              
    home/lex.tmux-local-socket-remote-machine                
    > ye53nis@ara-login01.rz.uni-jena.de's password:            
  2. first, connect with the GPU node in the high performance cluster
    cd /
    srun -p gpu_p100 --time=7-10:00:00 --ntasks-per-node=12 --mem-per-cpu=4000 --gres=gpu:1 --pty bash
    
    (base) [ye53nis@node128 /]$
    
  3. Load CUDA and cuDNN in the version compatible to your tensorflow library (see https://www.tensorflow.org/install/source#gpu)
    module load nvidia/cuda/11.2
    module load nvidia/cudnn/8.1
    module list
    
    Currently Loaded Modulefiles:
      1) nvidia/cuda/11.2   2) nvidia/cudnn/8.1
    (base) [ye53nis@node128 /]$
    
  4. Branch out git branch exp-210807-hparams from main (done via magit) and make sure you are on the correct branch
    cd /beegfs/ye53nis/drmed-git
    git checkout exp-220227-unet
    
    Checking out files: 100% (147/147), done.
    M       src/nanosimpy
    Branch exp-220227-unet set up to track remote branch exp-220227-unet from origin.
    Switched to a new branch 'exp-220227-unet'
    (base) [ye53nis@node128 drmed-git]$
    
  5. load conda environment, define MLflow environment variables and create log directory
    conda activate tf
    cd /beegfs/ye53nis/drmed-git
    export MLFLOW_EXPERIMENT_NAME=exp-220227-unet
    export MLFLOW_TRACKING_URI=file:./data/mlruns
    mkdir -p data/exp-220227-unet/jupyter
    mkdir ../tmp
    
    (tf) [ye53nis@node128 drmed-git]$
    
  6. set output directory for matplotlib plots in jupyter. Give this block the name jupyter-set-output-directory to be able to easily call it later.
    (setq org-babel-jupyter-resource-directory "./data/exp-220227-unet/jupyter")
    
    ./data/exp-220227-unet/jupyter
    

2.6.2 Setup: Jupyter node on HPC

  1. Set up tmux (if we haven't done that before) (#+CALL: setup-tmux[:session local])
         
    sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:
    > ye53nis@ara-login01.rz.uni-jena.de's password:
  2. Request compute node
    cd /
    srun -p b_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
    
  1. Start Jupyter Lab (#+CALL: jpt-tmux[:session jpmux])
    (tf) [ye53nis@node005 /]$ jupyter lab --no-browser --port=$PORT
    [I 2023-01-18 11:33:10.992 ServerApp] jupyterlab | extension was successfully linked.
    [I 2023-01-18 11:33:20.469 ServerApp] nbclassic | extension was successfully linked.
    [I 2023-01-18 11:33:20.932 ServerApp] nbclassic | extension was successfully loaded.
    [I 2023-01-18 11:33:20.935 LabApp] JupyterLab extension loaded from /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/jupyterlab
    [I 2023-01-18 11:33:20.935 LabApp] JupyterLab application directory is /home/ye53nis/.conda/envs/tf/share/jupyter/lab
    [I 2023-01-18 11:33:20.943 ServerApp] jupyterlab | extension was successfully loaded.
    [I 2023-01-18 11:33:20.945 ServerApp] Serving notebooks from local directory: /
    [I 2023-01-18 11:33:20.945 ServerApp] Jupyter Server 1.13.5 is running at:
    [I 2023-01-18 11:33:20.945 ServerApp] http://localhost:8889/lab?token=f8a5db2d00721ed0a736f6a6fc2a21020172913cc2337ec0
    [I 2023-01-18 11:33:20.945 ServerApp]  or http://127.0.0.1:8889/lab?token=f8a5db2d00721ed0a736f6a6fc2a21020172913cc2337ec0
    [I 2023-01-18 11:33:20.945 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
    [C 2023-01-18 11:33:21.003 ServerApp]
    
        To access the server, open this file in a browser:
            file:///home/ye53nis/.local/share/jupyter/runtime/jpserver-183673-open.html
        Or copy and paste one of these URLs:
            http://localhost:8889/lab?token=f8a5db2d00721ed0a736f6a6fc2a21020172913cc2337ec0
         or http://127.0.0.1:8889/lab?token=f8a5db2d00721ed0a736f6a6fc2a21020172913cc2337ec0
    
    
  2. Create SSH Tunnel for jupyter lab to the local computer (e.g. #+CALL: ssh-tunnel(port="8889", node="node160"))
                         
    sh-5.1$ sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:              
    Warning: Permanently added 'node005,192.168.193.5' (ECDSA) to the list of known hosts.
    ye53nis@node005's password:                  
  3. I started a Python3 kernel using jupyter-server-list-kernels. Then I added the kernel ID to the :PROPERTIES: drawer of this (and following) subtrees.
    python3           c4f3acce-60c4-489d-922c-407da110fd6a   a few seconds ago    idle       1
    
  4. Test (#+CALL: jp-metadata(_long='True)) and record metadata:

    No of CPUs in system: 48
    No of CPUs the current process can use: 24
    load average: (24.2, 19.65, 11.23)
    os.uname():  posix.uname_result(sysname='Linux', nodename='node034', release='3.10.0-957.1.3.el7.x86_64', version='#1 SMP Thu Nov 29 14:49:43 UTC 2018', machine='x86_64')
    PID of process: 102013
    RAM total: 137G, RAM used: 34G, RAM free: 64G
    the current directory: /
    My disk usage:
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/sda1             50G  4.9G   46G  10% /
    devtmpfs              63G     0   63G   0% /dev
    tmpfs                 63G  707M   63G   2% /dev/shm
    tmpfs                 63G  107M   63G   1% /run
    tmpfs                 63G     0   63G   0% /sys/fs/cgroup
    nfs01-ib:/home        80T   71T  9.2T  89% /home
    nfs03-ib:/pool/work  100T   72T   29T  72% /nfsdata
    nfs01-ib:/cluster    2.0T  486G  1.6T  24% /cluster
    /dev/sda5            2.0G   34M  2.0G   2% /tmp
    /dev/sda6            169G  4.0G  165G   3% /local
    /dev/sda3            6.0G  438M  5.6G   8% /var
    beegfs_nodev         524T  441T   84T  85% /beegfs
    tmpfs                 13G     0   13G   0% /run/user/67339# packages in environment at /home/ye53nis/.conda/envs/tf:
    #
    # Name                    Version                   Build  Channel
    _libgcc_mutex             0.1                        main
    _openmp_mutex             5.1                       1_gnu
    absl-py                   1.0.0                    pypi_0    pypi
    alembic                   1.7.7                    pypi_0    pypi
    anyio                     3.5.0            py39h06a4308_0
    argon2-cffi               21.3.0             pyhd3eb1b0_0
    argon2-cffi-bindings      21.2.0           py39h7f8727e_0
    asteval                   0.9.26                   pypi_0    pypi
    asttokens                 2.0.5              pyhd3eb1b0_0
    astunparse                1.6.3                    pypi_0    pypi
    attrs                     21.4.0             pyhd3eb1b0_0
    babel                     2.9.1              pyhd3eb1b0_0
    backcall                  0.2.0              pyhd3eb1b0_0
    beautifulsoup4            4.11.1           py39h06a4308_0
    bleach                    4.1.0              pyhd3eb1b0_0
    brotlipy                  0.7.0           py39h27cfd23_1003
    ca-certificates           2022.4.26            h06a4308_0
    cachetools                5.1.0                    pypi_0    pypi
    certifi                   2021.10.8        py39h06a4308_2
    cffi                      1.15.0           py39hd667e15_1
    charset-normalizer        2.0.4              pyhd3eb1b0_0
    click                     8.1.3                    pypi_0    pypi
    cloudpickle               2.0.0                    pypi_0    pypi
    cryptography              37.0.1           py39h9ce1e76_0
    cycler                    0.11.0                   pypi_0    pypi
    cython                    0.29.30                  pypi_0    pypi
    databricks-cli            0.16.6                   pypi_0    pypi
    debugpy                   1.5.1            py39h295c915_0
    decorator                 5.1.1              pyhd3eb1b0_0
    defusedxml                0.7.1              pyhd3eb1b0_0
    docker                    5.0.3                    pypi_0    pypi
    entrypoints               0.4              py39h06a4308_0
    executing                 0.8.3              pyhd3eb1b0_0
    fcsfiles                  2022.2.2                 pypi_0    pypi
    flask                     2.1.2                    pypi_0    pypi
    flatbuffers               1.12                     pypi_0    pypi
    fonttools                 4.33.3                   pypi_0    pypi
    future                    0.18.2                   pypi_0    pypi
    gast                      0.4.0                    pypi_0    pypi
    gitdb                     4.0.9                    pypi_0    pypi
    gitpython                 3.1.27                   pypi_0    pypi
    google-auth               2.6.6                    pypi_0    pypi
    google-auth-oauthlib      0.4.6                    pypi_0    pypi
    google-pasta              0.2.0                    pypi_0    pypi
    greenlet                  1.1.2                    pypi_0    pypi
    grpcio                    1.46.1                   pypi_0    pypi
    gunicorn                  20.1.0                   pypi_0    pypi
    h5py                      3.6.0                    pypi_0    pypi
    idna                      3.3                pyhd3eb1b0_0
    importlib-metadata        4.11.3                   pypi_0    pypi
    ipykernel                 6.9.1            py39h06a4308_0
    ipython                   8.3.0            py39h06a4308_0
    ipython_genutils          0.2.0              pyhd3eb1b0_1
    itsdangerous              2.1.2                    pypi_0    pypi
    jedi                      0.18.1           py39h06a4308_1
    jinja2                    3.0.3              pyhd3eb1b0_0
    joblib                    1.1.0                    pypi_0    pypi
    json5                     0.9.6              pyhd3eb1b0_0
    jsonschema                4.4.0            py39h06a4308_0
    jupyter_client            7.2.2            py39h06a4308_0
    jupyter_core              4.10.0           py39h06a4308_0
    jupyter_server            1.13.5             pyhd3eb1b0_0
    jupyterlab                3.3.2              pyhd3eb1b0_0
    jupyterlab_pygments       0.1.2                      py_0
    jupyterlab_server         2.12.0           py39h06a4308_0
    keras                     2.9.0                    pypi_0    pypi
    keras-preprocessing       1.1.2                    pypi_0    pypi
    kiwisolver                1.4.2                    pypi_0    pypi
    ld_impl_linux-64          2.38                 h1181459_0
    libclang                  14.0.1                   pypi_0    pypi
    libffi                    3.3                  he6710b0_2
    libgcc-ng                 11.2.0               h1234567_0
    libgomp                   11.2.0               h1234567_0
    libsodium                 1.0.18               h7b6447c_0
    libstdcxx-ng              11.2.0               h1234567_0
    lmfit                     1.0.3                    pypi_0    pypi
    mako                      1.2.0                    pypi_0    pypi
    markdown                  3.3.7                    pypi_0    pypi
    markupsafe                2.0.1            py39h27cfd23_0
    matplotlib                3.5.2                    pypi_0    pypi
    matplotlib-inline         0.1.2              pyhd3eb1b0_2
    mistune                   0.8.4           py39h27cfd23_1000
    mlflow                    1.26.0                   pypi_0    pypi
    multipletau               0.3.3                    pypi_0    pypi
    nbclassic                 0.3.5              pyhd3eb1b0_0
    nbclient                  0.5.13           py39h06a4308_0
    nbconvert                 6.4.4            py39h06a4308_0
    nbformat                  5.3.0            py39h06a4308_0
    ncurses                   6.3                  h7f8727e_2
    nest-asyncio              1.5.5            py39h06a4308_0
    notebook                  6.4.11           py39h06a4308_0
    numpy                     1.22.3                   pypi_0    pypi
    oauthlib                  3.2.0                    pypi_0    pypi
    openssl                   1.1.1o               h7f8727e_0
    opt-einsum                3.3.0                    pypi_0    pypi
    packaging                 21.3               pyhd3eb1b0_0
    pandas                    1.4.2                    pypi_0    pypi
    pandocfilters             1.5.0              pyhd3eb1b0_0
    parso                     0.8.3              pyhd3eb1b0_0
    pexpect                   4.8.0              pyhd3eb1b0_3
    pickleshare               0.7.5           pyhd3eb1b0_1003
    pillow                    9.1.1                    pypi_0    pypi
    pip                       21.2.4           py39h06a4308_0
    prometheus-flask-exporter 0.20.1                   pypi_0    pypi
    prometheus_client         0.13.1             pyhd3eb1b0_0
    prompt-toolkit            3.0.20             pyhd3eb1b0_0
    protobuf                  3.20.1                   pypi_0    pypi
    ptyprocess                0.7.0              pyhd3eb1b0_2
    pure_eval                 0.2.2              pyhd3eb1b0_0
    pyasn1                    0.4.8                    pypi_0    pypi
    pyasn1-modules            0.2.8                    pypi_0    pypi
    pycparser                 2.21               pyhd3eb1b0_0
    pygments                  2.11.2             pyhd3eb1b0_0
    pyjwt                     2.4.0                    pypi_0    pypi
    pyopenssl                 22.0.0             pyhd3eb1b0_0
    pyparsing                 3.0.4              pyhd3eb1b0_0
    pyrsistent                0.18.0           py39heee7806_0
    pysocks                   1.7.1            py39h06a4308_0
    python                    3.9.12               h12debd9_0
    python-dateutil           2.8.2              pyhd3eb1b0_0
    python-fastjsonschema     2.15.1             pyhd3eb1b0_0
    pytz                      2021.3             pyhd3eb1b0_0
    pyyaml                    6.0                      pypi_0    pypi
    pyzmq                     22.3.0           py39h295c915_2
    querystring-parser        1.2.4                    pypi_0    pypi
    readline                  8.1.2                h7f8727e_1
    requests                  2.27.1             pyhd3eb1b0_0
    requests-oauthlib         1.3.1                    pypi_0    pypi
    rsa                       4.8                      pypi_0    pypi
    scikit-learn              1.1.0                    pypi_0    pypi
    scipy                     1.8.1                    pypi_0    pypi
    seaborn                   0.11.2                   pypi_0    pypi
    send2trash                1.8.0              pyhd3eb1b0_1
    setuptools                61.2.0           py39h06a4308_0
    six                       1.16.0             pyhd3eb1b0_1
    smmap                     5.0.0                    pypi_0    pypi
    sniffio                   1.2.0            py39h06a4308_1
    soupsieve                 2.3.1              pyhd3eb1b0_0
    sqlalchemy                1.4.36                   pypi_0    pypi
    sqlite                    3.38.3               hc218d9a_0
    sqlparse                  0.4.2                    pypi_0    pypi
    stack_data                0.2.0              pyhd3eb1b0_0
    tabulate                  0.8.9                    pypi_0    pypi
    tensorboard               2.9.0                    pypi_0    pypi
    tensorboard-data-server   0.6.1                    pypi_0    pypi
    tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
    tensorflow                2.9.0                    pypi_0    pypi
    tensorflow-estimator      2.9.0                    pypi_0    pypi
    tensorflow-io-gcs-filesystem 0.26.0                   pypi_0    pypi
    termcolor                 1.1.0                    pypi_0    pypi
    terminado                 0.13.1           py39h06a4308_0
    testpath                  0.5.0              pyhd3eb1b0_0
    threadpoolctl             3.1.0                    pypi_0    pypi
    tk                        8.6.11               h1ccaba5_1
    tornado                   6.1              py39h27cfd23_0
    traitlets                 5.1.1              pyhd3eb1b0_0
    typing-extensions         4.1.1                hd3eb1b0_0
    typing_extensions         4.1.1              pyh06a4308_0
    tzdata                    2022a                hda174b7_0
    uncertainties             3.1.6                    pypi_0    pypi
    urllib3                   1.26.9           py39h06a4308_0
    wcwidth                   0.2.5              pyhd3eb1b0_0
    webencodings              0.5.1            py39h06a4308_1
    websocket-client          0.58.0           py39h06a4308_4
    werkzeug                  2.1.2                    pypi_0    pypi
    wheel                     0.37.1             pyhd3eb1b0_0
    wrapt                     1.14.1                   pypi_0    pypi
    xz                        5.2.5                h7f8727e_1
    zeromq                    4.3.4                h2531618_0
    zipp                      3.8.0                    pypi_0    pypi
    zlib                      1.2.12               h7f8727e_2
    
    Note: you may need to restart the kernel to use updated packages.
    {'SLURM_CHECKPOINT_IMAGE_DIR': '/var/slurm/checkpoint',
     'SLURM_NODELIST': 'node034',
     'SLURM_JOB_NAME': 'bash',
     'XDG_SESSION_ID': '135386',
     'SLURMD_NODENAME': 'node034',
     'SLURM_TOPOLOGY_ADDR': 'node034',
     'SLURM_NTASKS_PER_NODE': '24',
     'HOSTNAME': 'login01',
     'SLURM_PRIO_PROCESS': '0',
     'SLURM_SRUN_COMM_PORT': '40968',
     'SHELL': '/bin/bash',
     'TERM': 'xterm-color',
     'SLURM_JOB_QOS': 'qstand',
     'SLURM_PTY_WIN_ROW': '48',
     'HISTSIZE': '1000',
     'TMPDIR': '/tmp',
     'SLURM_TOPOLOGY_ADDR_PATTERN': 'node',
     'SSH_CLIENT': '10.231.185.64 42170 22',
     'CONDA_SHLVL': '2',
     'CONDA_PROMPT_MODIFIER': '(tf) ',
     'WINDOWID': '0',
     'QTDIR': '/usr/lib64/qt-3.3',
     'QTINC': '/usr/lib64/qt-3.3/include',
     'SSH_TTY': '/dev/pts/19',
     'NO_PROXY': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001',
     'QT_GRAPHICSSYSTEM_CHECKED': '1',
     'SLURM_NNODES': '1',
     'USER': 'ye53nis',
     'http_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
     'CONDA_EXE': '/cluster/miniconda3/bin/conda',
     'SLURM_STEP_NUM_NODES': '1',
     'SLURM_JOBID': '1657237',
     'SRUN_DEBUG': '3',
     'FTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'ftp_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_NTASKS': '24',
     'SLURM_LAUNCH_NODE_IPADDR': '192.168.192.5',
     'SLURM_STEP_ID': '0',
     'TMUX': '/tmp/tmux-67339/default,14861,2',
     '_CE_CONDA': '',
     'CONDA_PREFIX_1': '/cluster/miniconda3',
     'SLURM_STEP_LAUNCHER_PORT': '40968',
     'SLURM_TASKS_PER_NODE': '24',
     'MAIL': '/var/spool/mail/ye53nis',
     'PATH': '/home/ye53nis/.conda/envs/tf/bin:/home/lex/Programme/miniconda3/envs/tf/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/home/lex/Programme/miniconda3/bin:/usr/sbin:/home/ye53nis/.local/bin:/home/ye53nis/bin',
     'SLURM_WORKING_CLUSTER': 'hpc:192.168.192.1:6817:8448',
     'SLURM_JOB_ID': '1657237',
     'CONDA_PREFIX': '/home/ye53nis/.conda/envs/tf',
     'SLURM_JOB_USER': 'ye53nis',
     'SLURM_STEPID': '0',
     'PWD': '/',
     'SLURM_SRUN_COMM_HOST': '192.168.192.5',
     'LANG': 'en_US.UTF-8',
     'SLURM_PTY_WIN_COL': '236',
     'SLURM_UMASK': '0022',
     'MODULEPATH': '/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles',
     'SLURM_JOB_UID': '67339',
     'LOADEDMODULES': '',
     'SLURM_NODEID': '0',
     'TMUX_PANE': '%2',
     'SLURM_SUBMIT_DIR': '/',
     'SLURM_TASK_PID': '100551',
     'SLURM_NPROCS': '24',
     'SLURM_CPUS_ON_NODE': '24',
     'SLURM_DISTRIBUTION': 'block',
     'HTTPS_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'https_proxy': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_PROCID': '0',
     'HISTCONTROL': 'ignoredups',
     '_CE_M': '',
     'SLURM_JOB_NODELIST': 'node034',
     'SLURM_PTY_PORT': '43329',
     'HOME': '/home/ye53nis',
     'SHLVL': '3',
     'SLURM_LOCALID': '0',
     'SLURM_JOB_GID': '13280',
     'SLURM_JOB_CPUS_PER_NODE': '24',
     'SLURM_CLUSTER_NAME': 'hpc',
     'no_proxy': 'localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001',
     'SLURM_GTIDS': '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23',
     'SLURM_SUBMIT_HOST': 'login01',
     'HTTP_PROXY': 'http://internet4nzm.rz.uni-jena.de:3128',
     'SLURM_JOB_PARTITION': 'b_standard',
     'MATHEMATICA_HOME': '/cluster/apps/mathematica/12.3',
     'CONDA_PYTHON_EXE': '/cluster/miniconda3/bin/python',
     'LOGNAME': 'ye53nis',
     'SLURM_STEP_NUM_TASKS': '24',
     'QTLIB': '/usr/lib64/qt-3.3/lib',
     'SLURM_JOB_ACCOUNT': 'iaob',
     'SLURM_JOB_NUM_NODES': '1',
     'MODULESHOME': '/usr/share/Modules',
     'CONDA_DEFAULT_ENV': 'tf',
     'LESSOPEN': '||/usr/bin/lesspipe.sh %s',
     'SLURM_STEP_TASKS_PER_NODE': '24',
     'PORT': '8889',
     'SLURM_STEP_NODELIST': 'node034',
     'DISPLAY': ':0',
     'XDG_RUNTIME_DIR': '',
     'XAUTHORITY': '/home/lex/.Xauthority',
     'BASH_FUNC_module()': '() {  eval `/usr/bin/modulecmd bash $*`\n}',
     '_': '/home/ye53nis/.conda/envs/tf/bin/jupyter',
     'PYDEVD_USE_FRAME_EVAL': 'NO',
     'JPY_PARENT_PID': '100629',
     'CLICOLOR': '1',
     'PAGER': 'cat',
     'GIT_PAGER': 'cat',
     'MPLBACKEND': 'module://matplotlib_inline.backend_inline'}
    

2.6.3 Setup: Jupyter on local computer

  1. on our local machine we don't need tmux. A simple sh command is enough. So let's start the conda environment in the sh session local and start jupterlab there.
    conda activate tf
    jupyter lab --no-browser --port=8888
    
    [I 2022-05-22 14:24:33.496 ServerApp] jupyterlab | extension was successfully linked.
    [I 2022-05-22 14:24:33.745 ServerApp] nbclassic | extension was successfully linked.
    [I 2022-05-22 14:24:33.786 LabApp] JupyterLab extension loaded from /home/lex/Programme/miniconda3/envs/tf/lib/python3.9/site-packages/jupyterlab
    [I 2022-05-22 14:24:33.786 LabApp] JupyterLab application directory is /home/lex/Programme/miniconda3/envs/tf/share/jupyter/lab
    [I 2022-05-22 14:24:33.790 ServerApp] jupyterlab | extension was successfully loaded.
    [I 2022-05-22 14:24:33.799 ServerApp] nbclassic | extension was successfully loaded.
    [I 2022-05-22 14:24:33.800 ServerApp] Serving notebooks from local directory: /home/lex/Programme/drmed-git
    [I 2022-05-22 14:24:33.800 ServerApp] Jupyter Server 1.4.1 is running at:
    [I 2022-05-22 14:24:33.800 ServerApp] http://localhost:8888/lab?token=1d0362f7e2280b0060620c901abee258910e16c879bc0870
    [I 2022-05-22 14:24:33.800 ServerApp]  or http://127.0.0.1:8888/lab?token=1d0362f7e2280b0060620c901abee258910e16c879bc0870
    [I 2022-05-22 14:24:33.800 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
    [C 2022-05-22 14:24:33.804 ServerApp]
    
        To access the server, open this file in a browser:
            file:///home/lex/.local/share/jupyter/runtime/jpserver-1749996-open.html
        Or copy and paste one of these URLs:
            http://localhost:8888/lab?token=1d0362f7e2280b0060620c901abee258910e16c879bc0870
         or http://127.0.0.1:8888/lab?token=1d0362f7e2280b0060620c901abee258910e16c879bc0870
    
    
  2. I started a Python3 kernel using jupyter-server-list-kernels. Then I added the kernel ID to the :PROPERTIES: drawer of this (and following) subtrees.
    python3           03038b73-b2b5-49ce-a1dc-21afb6247d0f   a few seconds ago    starting   0
    
  3. Test: (#+CALL: jp-metadata(_long='True))
    No of CPUs in system: 4
    No of CPUs the current process can use: 4
    load average: (0.93115234375, 0.97216796875, 0.5595703125)
    os.uname():  posix.uname_result(sysname='Linux', nodename='Topialex', release='5.15.28-1-MANJARO', version='#1 SMP PREEMPT Fri Mar 11 14:12:57 UTC 2022', machine='x86_64')
    PID of process: 8991
    RAM total: 16Gi, RAM used: 1,8Gi, RAM free: 12Gi
    the current directory: /home/lex/Programme/drmed-git
    My disk usage:
    Filesystem      Size  Used Avail Use% Mounted on
    dev             3,9G     0  3,9G   0% /dev
    run             3,9G  1,5M  3,9G   1% /run
    /dev/sda2       167G  131G   28G  83% /
    tmpfs           3,9G   63M  3,8G   2% /dev/shm
    tmpfs           3,9G  4,2M  3,9G   1% /tmp
    /dev/sda1       300M  264K  300M   1% /boot/efi
    tmpfs           784M   80K  784M   1% /run/user/1000# packages in environment at /home/lex/Programme/miniconda3/envs/tf:
    #
    # Name                    Version                   Build  Channel
    _libgcc_mutex             0.1                        main
    _openmp_mutex             4.5                       1_gnu
    absl-py                   1.0.0                    pypi_0    pypi
    alembic                   1.4.1                    pypi_0    pypi
    anyio                     2.2.0            py39h06a4308_1
    argon2-cffi               20.1.0           py39h27cfd23_1
    asteval                   0.9.25                   pypi_0    pypi
    astroid                   2.9.2                    pypi_0    pypi
    astunparse                1.6.3                    pypi_0    pypi
    async_generator           1.10               pyhd3eb1b0_0
    attrs                     21.2.0             pyhd3eb1b0_0
    babel                     2.9.1              pyhd3eb1b0_0
    backcall                  0.2.0              pyhd3eb1b0_0
    bleach                    4.0.0              pyhd3eb1b0_0
    brotlipy                  0.7.0           py39h27cfd23_1003
    ca-certificates           2021.10.26           h06a4308_2
    cachetools                4.2.4                    pypi_0    pypi
    certifi                   2021.10.8        py39h06a4308_0
    cffi                      1.14.6           py39h400218f_0
    charset-normalizer        2.0.4              pyhd3eb1b0_0
    click                     8.0.3                    pypi_0    pypi
    cloudpickle               2.0.0                    pypi_0    pypi
    cryptography              36.0.0           py39h9ce1e76_0
    cycler                    0.11.0                   pypi_0    pypi
    cython                    0.29.26                  pypi_0    pypi
    databricks-cli            0.16.2                   pypi_0    pypi
    debugpy                   1.5.1            py39h295c915_0
    decorator                 5.1.0              pyhd3eb1b0_0
    defusedxml                0.7.1              pyhd3eb1b0_0
    docker                    5.0.3                    pypi_0    pypi
    entrypoints               0.3              py39h06a4308_0
    fcsfiles                  2021.6.6                 pypi_0    pypi
    flake8                    4.0.1                    pypi_0    pypi
    flask                     2.0.2                    pypi_0    pypi
    flatbuffers               2.0                      pypi_0    pypi
    focuspoint                0.1                      pypi_0    pypi
    fonttools                 4.28.5                   pypi_0    pypi
    future                    0.18.2                   pypi_0    pypi
    gast                      0.4.0                    pypi_0    pypi
    gitdb                     4.0.9                    pypi_0    pypi
    gitpython                 3.1.24                   pypi_0    pypi
    google-auth               2.3.3                    pypi_0    pypi
    google-auth-oauthlib      0.4.6                    pypi_0    pypi
    google-pasta              0.2.0                    pypi_0    pypi
    greenlet                  1.1.2                    pypi_0    pypi
    grpcio                    1.43.0                   pypi_0    pypi
    gunicorn                  20.1.0                   pypi_0    pypi
    h5py                      3.6.0                    pypi_0    pypi
    idna                      3.3                pyhd3eb1b0_0
    importlib-metadata        4.8.2            py39h06a4308_0
    importlib_metadata        4.8.2                hd3eb1b0_0
    ipykernel                 6.4.1            py39h06a4308_1
    ipython                   7.29.0           py39hb070fc8_0
    ipython_genutils          0.2.0              pyhd3eb1b0_1
    isort                     5.10.1                   pypi_0    pypi
    itsdangerous              2.0.1                    pypi_0    pypi
    jedi                      0.18.0           py39h06a4308_1
    jinja2                    3.0.2              pyhd3eb1b0_0
    joblib                    1.1.0                    pypi_0    pypi
    json5                     0.9.6              pyhd3eb1b0_0
    jsonschema                3.2.0              pyhd3eb1b0_2
    jupyter_client            7.1.0              pyhd3eb1b0_0
    jupyter_core              4.9.1            py39h06a4308_0
    jupyter_server            1.4.1            py39h06a4308_0
    jupyterlab                3.2.1              pyhd3eb1b0_1
    jupyterlab_pygments       0.1.2                      py_0
    jupyterlab_server         2.8.2              pyhd3eb1b0_0
    keras                     2.7.0                    pypi_0    pypi
    keras-preprocessing       1.1.2                    pypi_0    pypi
    kiwisolver                1.3.2                    pypi_0    pypi
    lazy-object-proxy         1.7.1                    pypi_0    pypi
    ld_impl_linux-64          2.35.1               h7274673_9
    libclang                  12.0.0                   pypi_0    pypi
    libffi                    3.3                  he6710b0_2
    libgcc-ng                 9.3.0               h5101ec6_17
    libgomp                   9.3.0               h5101ec6_17
    libsodium                 1.0.18               h7b6447c_0
    libstdcxx-ng              9.3.0               hd4cf53a_17
    lmfit                     1.0.3                    pypi_0    pypi
    mako                      1.1.6                    pypi_0    pypi
    markdown                  3.3.6                    pypi_0    pypi
    markupsafe                2.0.1            py39h27cfd23_0
    matplotlib                3.5.1                    pypi_0    pypi
    matplotlib-inline         0.1.2              pyhd3eb1b0_2
    mccabe                    0.6.1                    pypi_0    pypi
    mistune                   0.8.4           py39h27cfd23_1000
    mlflow                    1.22.0                   pypi_0    pypi
    multipletau               0.3.3                    pypi_0    pypi
    mypy                      0.930                    pypi_0    pypi
    mypy-extensions           0.4.3                    pypi_0    pypi
    nbclassic                 0.2.6              pyhd3eb1b0_0
    nbclient                  0.5.3              pyhd3eb1b0_0
    nbconvert                 6.1.0            py39h06a4308_0
    nbformat                  5.1.3              pyhd3eb1b0_0
    ncurses                   6.3                  h7f8727e_2
    nest-asyncio              1.5.1              pyhd3eb1b0_0
    nodeenv                   1.6.0                    pypi_0    pypi
    notebook                  6.4.6            py39h06a4308_0
    numpy                     1.21.5                   pypi_0    pypi
    oauthlib                  3.1.1                    pypi_0    pypi
    openssl                   1.1.1l               h7f8727e_0
    opt-einsum                3.3.0                    pypi_0    pypi
    packaging                 21.3               pyhd3eb1b0_0
    pandas                    1.3.5                    pypi_0    pypi
    pandocfilters             1.4.3            py39h06a4308_1
    parso                     0.8.2              pyhd3eb1b0_0
    pexpect                   4.8.0              pyhd3eb1b0_3
    pickleshare               0.7.5           pyhd3eb1b0_1003
    pillow                    8.4.0                    pypi_0    pypi
    pip                       21.2.4           py39h06a4308_0
    platformdirs              2.4.1                    pypi_0    pypi
    prometheus-flask-exporter 0.18.7                   pypi_0    pypi
    prometheus_client         0.12.0             pyhd3eb1b0_0
    prompt-toolkit            3.0.20             pyhd3eb1b0_0
    protobuf                  3.19.1                   pypi_0    pypi
    ptyprocess                0.7.0              pyhd3eb1b0_2
    pyasn1                    0.4.8                    pypi_0    pypi
    pyasn1-modules            0.2.8                    pypi_0    pypi
    pycodestyle               2.8.0                    pypi_0    pypi
    pycparser                 2.21               pyhd3eb1b0_0
    pydot                     1.4.2                    pypi_0    pypi
    pyflakes                  2.4.0                    pypi_0    pypi
    pygments                  2.10.0             pyhd3eb1b0_0
    pylint                    2.12.2                   pypi_0    pypi
    pyopenssl                 21.0.0             pyhd3eb1b0_1
    pyparsing                 3.0.4              pyhd3eb1b0_0
    pyright                   0.0.13                   pypi_0    pypi
    pyrsistent                0.18.0           py39heee7806_0
    pysocks                   1.7.1            py39h06a4308_0
    python                    3.9.7                h12debd9_1
    python-dateutil           2.8.2              pyhd3eb1b0_0
    python-editor             1.0.4                    pypi_0    pypi
    pytz                      2021.3             pyhd3eb1b0_0
    pyyaml                    6.0                      pypi_0    pypi
    pyzmq                     22.3.0           py39h295c915_2
    querystring-parser        1.2.4                    pypi_0    pypi
    readline                  8.1                  h27cfd23_0
    requests                  2.26.0             pyhd3eb1b0_0
    requests-oauthlib         1.3.0                    pypi_0    pypi
    rsa                       4.8                      pypi_0    pypi
    scikit-learn              1.0.2                    pypi_0    pypi
    scipy                     1.7.3                    pypi_0    pypi
    seaborn                   0.11.2                   pypi_0    pypi
    send2trash                1.8.0              pyhd3eb1b0_1
    setuptools                58.0.4           py39h06a4308_0
    six                       1.16.0             pyhd3eb1b0_0
    smmap                     5.0.0                    pypi_0    pypi
    sniffio                   1.2.0            py39h06a4308_1
    sqlalchemy                1.4.29                   pypi_0    pypi
    sqlite                    3.37.0               hc218d9a_0
    sqlparse                  0.4.2                    pypi_0    pypi
    tabulate                  0.8.9                    pypi_0    pypi
    tensorboard               2.7.0                    pypi_0    pypi
    tensorboard-data-server   0.6.1                    pypi_0    pypi
    tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
    tensorflow                2.7.0                    pypi_0    pypi
    tensorflow-estimator      2.7.0                    pypi_0    pypi
    tensorflow-io-gcs-filesystem 0.23.1                   pypi_0    pypi
    termcolor                 1.1.0                    pypi_0    pypi
    terminado                 0.9.4            py39h06a4308_0
    testpath                  0.5.0              pyhd3eb1b0_0
    threadpoolctl             3.0.0                    pypi_0    pypi
    tk                        8.6.11               h1ccaba5_0
    toml                      0.10.2                   pypi_0    pypi
    tomli                     2.0.0                    pypi_0    pypi
    tornado                   6.1              py39h27cfd23_0
    traitlets                 5.1.1              pyhd3eb1b0_0
    typing-extensions         4.0.1                    pypi_0    pypi
    tzdata                    2021e                hda174b7_0
    uncertainties             3.1.6                    pypi_0    pypi
    urllib3                   1.26.7             pyhd3eb1b0_0
    wcwidth                   0.2.5              pyhd3eb1b0_0
    webencodings              0.5.1            py39h06a4308_1
    websocket-client          1.2.3                    pypi_0    pypi
    werkzeug                  2.0.2                    pypi_0    pypi
    wheel                     0.37.0             pyhd3eb1b0_1
    wrapt                     1.13.3                   pypi_0    pypi
    xz                        5.2.5                h7b6447c_0
    zeromq                    4.3.4                h2531618_0
    zipp                      3.6.0              pyhd3eb1b0_0
    zlib                      1.2.11               h7f8727e_4
    
    Note: you may need to restart the kernel to use updated packages.
    {'SHELL': '/bin/bash',
     'SESSION_MANAGER': 'local/Topialex:@/tmp/.ICE-unix/878,unix/Topialex:/tmp/.ICE-unix/878',
     'XDG_CONFIG_DIRS': '/home/lex/.config/kdedefaults:/etc/xdg',
     'XDG_SESSION_PATH': '/org/freedesktop/DisplayManager/Session1',
     'CONDA_EXE': '/home/lex/Programme/miniconda3/bin/conda',
     '_CE_M': '',
     'LANGUAGE': 'en_GB',
     'TERMCAP': '',
     'LC_ADDRESS': 'de_DE.UTF-8',
     'LC_NAME': 'de_DE.UTF-8',
     'INSIDE_EMACS': '27.2,comint',
     'DESKTOP_SESSION': 'plasma',
     'LC_MONETARY': 'de_DE.UTF-8',
     'GTK_RC_FILES': '/etc/gtk/gtkrc:/home/lex/.gtkrc:/home/lex/.config/gtkrc',
     'XCURSOR_SIZE': '24',
     'GTK_MODULES': 'canberra-gtk-module',
     'XDG_SEAT': 'seat0',
     'PWD': '/home/lex/Programme/drmed-git',
     'LOGNAME': 'lex',
     'XDG_SESSION_DESKTOP': 'KDE',
     'XDG_SESSION_TYPE': 'x11',
     'CONDA_PREFIX': '/home/lex/Programme/miniconda3/envs/tf',
     'DSSI_PATH': '/home/lex/.dssi:/usr/lib/dssi:/usr/local/lib/dssi',
     'SYSTEMD_EXEC_PID': '768',
     'XAUTHORITY': '/home/lex/.Xauthority',
     'MOTD_SHOWN': 'pam',
     'GTK2_RC_FILES': '/etc/gtk-2.0/gtkrc:/home/lex/.gtkrc-2.0:/home/lex/.config/gtkrc-2.0',
     'HOME': '/home/lex',
     'LANG': 'de_DE.UTF-8',
     'LC_PAPER': 'de_DE.UTF-8',
     'VST_PATH': '/home/lex/.vst:/usr/lib/vst:/usr/local/lib/vst',
     'XDG_CURRENT_DESKTOP': 'KDE',
     'COLUMNS': '80',
     'CONDA_PROMPT_MODIFIER': '',
     'XDG_SEAT_PATH': '/org/freedesktop/DisplayManager/Seat0',
     'KDE_SESSION_UID': '1000',
     'XDG_SESSION_CLASS': 'user',
     'LC_IDENTIFICATION': 'de_DE.UTF-8',
     'TERM': 'xterm-color',
     '_CE_CONDA': '',
     'USER': 'lex',
     'CONDA_SHLVL': '1',
     'KDE_SESSION_VERSION': '5',
     'PAM_KWALLET5_LOGIN': '/run/user/1000/kwallet5.socket',
     'DISPLAY': ':0',
     'SHLVL': '2',
     'LC_TELEPHONE': 'de_DE.UTF-8',
     'LC_MEASUREMENT': 'de_DE.UTF-8',
     'XDG_VTNR': '1',
     'XDG_SESSION_ID': '2',
     'QT_LINUX_ACCESSIBILITY_ALWAYS_ON': '1',
     'CONDA_PYTHON_EXE': '/home/lex/Programme/miniconda3/bin/python',
     'MOZ_PLUGIN_PATH': '/usr/lib/mozilla/plugins',
     'XDG_RUNTIME_DIR': '/run/user/1000',
     'CONDA_DEFAULT_ENV': 'tf',
     'LC_TIME': 'de_DE.UTF-8',
     'QT_AUTO_SCREEN_SCALE_FACTOR': '0',
     'XCURSOR_THEME': 'breeze_cursors',
     'XDG_DATA_DIRS': '/home/lex/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share:/var/lib/snapd/desktop',
     'KDE_FULL_SESSION': 'true',
     'BROWSER': 'vivaldi-stable',
     'PATH': '/home/lex/Programme/miniconda3/envs/tf/bin:/home/lex/Programme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin',
     'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus',
     'LV2_PATH': '/home/lex/.lv2:/usr/lib/lv2:/usr/local/lib/lv2',
    'KDE_APPLICATIONS_AS_SCOPE': '1',
    'MAIL': '/var/spool/mail/lex',
    'LC_NUMERIC': 'de_DE.UTF-8',
    'LADSPA_PATH': '/home/lex/.ladspa:/usr/lib/ladspa:/usr/local/lib/ladspa',
    'CADENCE_AUTO_STARTED': 'true',
    '_': '/home/lex/Programme/miniconda3/envs/tf/bin/jupyter',
    'PYDEVD_USE_FRAME_EVAL': 'NO',
    'JPY_PARENT_PID': '8414',
    'CLICOLOR': '1',
    'PAGER': 'cat',
    'GIT_PAGER': 'cat',
    'MPLBACKEND': 'module://matplotlib_inline.backend_inline'}
    
    5f083adb-6166-4a49-9fa5-08e37046cbfd
    

2.6.4 Setup: Node for running Mlflow UI

  1. Create mlflow tmux session and start mlflow ui
    conda activate tf
    mlflow ui --backend-store-uri file:///beegfs/ye53nis/drmed-git/data/mlruns -p 5001
    
    (tf) [ye53nis@login01 ~]$ mlflow ui --backend-store-uri file:///beegfs/ye53nis/drmed-git/data/mlruns -p 5001
    [2021-08-08 14:47:33 +0200] [5106] [INFO] Starting gunicorn 20.1.0
    [2021-08-08 14:47:33 +0200] [5106] [INFO] Listening at: http://127.0.0.1:5001 (5106)
    [2021-08-08 14:47:33 +0200] [5106] [INFO] Using worker: sync
    [2021-08-08 14:47:33 +0200] [5115] [INFO] Booting worker with pid: 5115
    
  2. SHH tunnel the mflow session to the local computer (#+CALL: ssh-tunnel[:session local3](port="5001", node="login01"))
                     
    sh-5.1$ sh-5.1$
    ye53nis@ara-login01.rz.uni-jena.de's password:          
    ye53nis@login01's password:              
    bind: Address already in use
           
    Last login: Tue Aug 17 18:03:52 2021 from 10.231.188.20

2.6.5 Setup: Record GPU metadata & git

  1. Current directory, last 10 git commits

    pwd
    git log -10
    
    (tf) [ye53nis@node128 drmed-git]$ pwd
    /beegfs/ye53nis/drmed-git
    (tf) [ye53nis@node128 drmed-git]$ git log -10
    commit 4c2dc79f0483090d3af2591891c2349b0a48115f
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Thu Mar 3 14:10:45 2022 +0100
    
        Fix normalize() for l1 and l2 in preprocessing
    
    commit 39baf02076ba8fbcf444bfa11108d302bcb4c45f
    Author: Alex Seltmann <seltmann@posteo.de>
    Date:   Sun Feb 27 22:20:22 2022 +0100
    
        Add comparison file from exp-210807-hparams
    
    commit d51b11eda090b9301e783ec35bdfd26c7bf0709c
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Sun Feb 27 18:40:00 2022 +0100
    
        fix model input_size to None; else to crop_size
    
    commit c637444d8b798603629f6f0bd72ee55af7f81a5f
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Sun Feb 27 18:39:29 2022 +0100
    
        Fix function call correlate_and_fit
    
    commit 291c6619c12bc39d526137a43d976b3cb4881e50
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Sat Feb 26 20:04:07 2022 +0100
    
        Fix scale_trace; simplify tf_pad_trace call
    
    commit dcca8b9e17909a95b824c8a7b1fec52eeed198c3
    Author: Apoplex <oligolex@vivaldi.net>
    Date:   Thu Feb 24 16:11:39 2022 +0100
    
        test tf_pad_trace
    
    commit 6cf2da85748ef13f2e752bea8989a6d31549ced3
    (tf) [ye53nis@node128 drmed-git]$
    
  2. GPU, CPU, RAM, file system, env variables, top info
    nvcc -V
    echo --------------------
    lscpu
    echo --------------------
    nproc
    echo --------------------
    free -h
    echo --------------------
    df -h
    echo --------------------
    printenv
    echo --------------------
    top -bcn1 -w512 | head -n 15
    
    (tf) [ye53nis@node128 drmed-git]$ nvcc -V
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2020 NVIDIA Corporation
    Built on Mon_Nov_30_19:08:53_PST_2020
    Cuda compilation tools, release 11.2, V11.2.67
    Build cuda_11.2.r11.2/compiler.29373293_0
    --------------------
    (tf) [ye53nis@node128 drmed-git]$ lscpu
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                48
    On-line CPU(s) list:   0-47
    Thread(s) per core:    2
    Core(s) per socket:    12
    Socket(s):             2
    NUMA node(s):          4
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 79
    Model name:            Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
    Stepping:              1
    CPU MHz:               1203.527
    CPU max MHz:           2900.0000
    CPU min MHz:           1200.0000
    BogoMIPS:              4399.79
    Virtualization:        VT-x
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              256K
    L3 cache:              15360K
    NUMA node0 CPU(s):     0-5,24-29
    NUMA node1 CPU(s):     6-11,30-35
    NUMA node2 CPU(s):     12-17,36-41
    NUMA node3 CPU(s):     18-23,42-47
    Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonst
    op_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cd
    p_l3 intel_ppin intel_pt ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a rdseed adx smap xsaveopt cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida
    arat pln pts spec_ctrl intel_stibp
    --------------------
    (tf) [ye53nis@node128 drmed-git]$ nproc
    12
    --------------------
    (tf) [ye53nis@node128 drmed-git]$ free -h
                  total        used        free      shared  buff/cache   available
    Mem:           125G        1.1G        116G        230M        8.6G        123G
    Swap:           11G          0B         11G
    --------------------
    (tf) [ye53nis@node128 drmed-git]$ df -h
    Filesystem           Size  Used Avail Use% Mounted on
    /dev/sda1             50G  7.0G   44G  14% /
    devtmpfs              63G     0   63G   0% /dev
    tmpfs                 63G  188M   63G   1% /dev/shm
    tmpfs                 63G   43M   63G   1% /run
    tmpfs                 63G     0   63G   0% /sys/fs/cgroup
    nfs01-ib:/cluster    2.0T  469G  1.6T  23% /cluster
    nfs01-ib:/home        80T   68T   13T  85% /home
    nfs03-ib:/pool/work  100T   71T   29T  71% /nfsdata
    /dev/sda3            6.0G  635M  5.4G  11% /var
    /dev/sda6            169G  354M  169G   1% /local
    /dev/sda5            2.0G   35M  2.0G   2% /tmp
    beegfs_nodev         524T  508T   17T  97% /beegfs
    --------------------
    (tf) [ye53nis@node128 drmed-git]$ printenv
    SLURM_CHECKPOINT_IMAGE_DIR=/var/slurm/checkpoint
    SLURM_NODELIST=node128
    CUDA_PATH=/cluster/nvidia/cuda/11.2
    SLURM_JOB_NAME=bash
    CUDA_INC_PATH=/cluster/nvidia/cuda/11.2/include
    XDG_SESSION_ID=44301
    SLURMD_NODENAME=node128
    SLURM_TOPOLOGY_ADDR=node128
    SLURM_NTASKS_PER_NODE=12
    HOSTNAME=login01
    SLURM_PRIO_PROCESS=0
    SLURM_SRUN_COMM_PORT=38740
    SHELL=/bin/bash
    TERM=screen
    MLFLOW_EXPERIMENT_NAME=exp-220227-unet
    SLURM_JOB_QOS=qstand
    SLURM_PTY_WIN_ROW=53
    HISTSIZE=1000
    TMPDIR=/tmp
    SLURM_TOPOLOGY_ADDR_PATTERN=node
    SSH_CLIENT=10.231.181.128 49370 22
    INCLUDEDIR=/cluster/nvidia/cuda/11.2/include
    CONDA_SHLVL=2
    CONDA_PROMPT_MODIFIER=(tf)
    OLDPWD=/beegfs/ye53nis/drmed-git
    QTDIR=/usr/lib64/qt-3.3
    QTINC=/usr/lib64/qt-3.3/include
    SSH_TTY=/dev/pts/79
    NO_PROXY=localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001
    QT_GRAPHICSSYSTEM_CHECKED=1
    SLURM_NNODES=1
    USER=ye53nis
    http_proxy=http://internet4nzm.rz.uni-jena.de:3128
    LD_LIBRARY_PATH=/cluster/nvidia/cuda/11.2/lib64:/cluster/nvidia/cuda/11.2/nvvm/lib64:/cluster/nvidia/cudnn/8.1//lib64
    LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01
    ;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:
    *.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01
    ;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:
    *.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;3
    5:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=
    01;36:*.xspf=01;36:
    CONDA_EXE=/cluster/miniconda3/bin/conda
    SLURM_STEP_NUM_NODES=1
    SLURM_JOBID=1615665
    SRUN_DEBUG=3
    FTP_PROXY=http://internet4nzm.rz.uni-jena.de:3128
    ftp_proxy=http://internet4nzm.rz.uni-jena.de:3128
    SLURM_NTASKS=12
    SLURM_LAUNCH_NODE_IPADDR=192.168.192.5
    SLURM_STEP_ID=0
    TMUX=/tmp/tmux-67339/default,20557,7
    _CE_CONDA=
    CONDA_PREFIX_1=/cluster/miniconda3
    MODCUDA=YES
    SLURM_STEP_LAUNCHER_PORT=38740
    SLURM_TASKS_PER_NODE=12
    MAIL=/var/spool/mail/ye53nis
    PATH=/cluster/nvidia/cuda/11.2/bin:/cluster/nvidia/cuda/11.2/nvvm:/cluster/nvidia/cuda/11.2/open64/bin:/cluster/nvidia/cuda/11.2/libnvvp:/home/ye53nis/.conda/envs/tf/bin:/home/lex/Programme/miniconda3/envs/tf-nightly-lab/bin:/home/lex/P
    rogramme/miniconda3/condabin:/home/lex/.local/bin:/bin:/usr/bin:/usr/local/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin:/usr/sbin:/home/ye53nis/.local/bi
    n:/home/ye53nis/bin
    SLURM_WORKING_CLUSTER=hpc:192.168.192.1:6817:8448
    SLURM_JOB_ID=1615665
    LD_RUN_PATH=/cluster/nvidia/cuda/11.2/lib64
    SLURM_STEP_GPUS=0
    CONDA_PREFIX=/home/ye53nis/.conda/envs/tf
    CUDA_LIB_PATH=/cluster/nvidia/cuda/11.2/lib64
    SLURM_JOB_USER=ye53nis
    SLURM_STEPID=0
    PWD=/beegfs/ye53nis/drmed-git
    _LMFILES_=/cluster/modulefiles/nvidia/cuda/11.2:/cluster/modulefiles/nvidia/cudnn/8.1
    CUDA_VISIBLE_DEVICES=0
    SLURM_SRUN_COMM_HOST=192.168.192.5
    LANG=en_US.UTF-8
    SLURM_PTY_WIN_COL=236
    SLURM_UMASK=0022
    MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles:/cluster/modulefiles
    SLURM_JOB_UID=67339
    LOADEDMODULES=nvidia/cuda/11.2:nvidia/cudnn/8.1
    SLURM_NODEID=0
    TMUX_PANE=%7
    SLURM_SUBMIT_DIR=/
    SLURM_TASK_PID=4042
    SLURM_NPROCS=12
    SLURM_CPUS_ON_NODE=12
    SLURM_DISTRIBUTION=block
    HTTPS_PROXY=http://internet4nzm.rz.uni-jena.de:3128
    https_proxy=http://internet4nzm.rz.uni-jena.de:3128
    SLURM_PROCID=0
    HISTCONTROL=ignoredups
    _CE_M=
    SLURM_JOB_NODELIST=node128
    SLURM_PTY_PORT=37529
    HOME=/home/ye53nis
    SHLVL=3
    SLURM_LOCALID=0
    SLURM_JOB_GID=13280
    SLURM_JOB_CPUS_PER_NODE=12
    SLURM_CLUSTER_NAME=hpc
    no_proxy=localhost,127.0.0.0/8,.uni-jena.de,141.35.0.0/16,10.0.0.0/8,192.168.0.0/16,172.0.0.0/8,fe80::/7,2001:638:1558::/24,vmaster,node001
    SLURM_GTIDS=0,1,2,3,4,5,6,7,8,9,10,11
    SLURM_SUBMIT_HOST=login01
    HTTP_PROXY=http://internet4nzm.rz.uni-jena.de:3128
    SLURM_JOB_PARTITION=gpu_p100
    MATHEMATICA_HOME=/cluster/apps/mathematica/11.3
    CONDA_PYTHON_EXE=/cluster/miniconda3/bin/python
    LOGNAME=ye53nis
    SLURM_STEP_NUM_TASKS=12
    QTLIB=/usr/lib64/qt-3.3/lib
    GPU_DEVICE_ORDINAL=0
    SLURM_JOB_ACCOUNT=iaob
    MLFLOW_TRACKING_URI=file:./data/mlruns
    SLURM_JOB_NUM_NODES=1
    MODULESHOME=/usr/share/Modules
    CONDA_DEFAULT_ENV=tf
    LESSOPEN=||/usr/bin/lesspipe.sh %s
    SLURM_STEP_TASKS_PER_NODE=12
    SLURM_STEP_NODELIST=node128
    DISPLAY=:0
    XDG_RUNTIME_DIR=/run/user/67339
    INCLUDE=/cluster/nvidia/cudnn/8.1//include
    XAUTHORITY=/home/lex/.Xauthority
    BASH_FUNC_module()=() {  eval `/usr/bin/modulecmd bash $*`
    }
    _=/bin/printenv
    --------------------
    (tf) [ye53nis@node128 drmed-git]$ top -bcn1 -w512 | head -n 15
    top - 21:21:42 up 72 days,  9:36,  0 users,  load average: 0.00, 0.03, 0.05
    Tasks: 521 total,   1 running, 520 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  0.2 us,  0.2 sy,  0.0 ni, 99.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem : 13191630+total, 12171446+free,  1196368 used,  9005476 buff/cache
    KiB Swap: 12582908 total, 12582908 free,        0 used. 12953688+avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    13258 ye53nis   20   0  172732   2620   1664 R  11.1  0.0   0:00.03 top -bcn1 -w512
        1 root      20   0   71788   7548   2584 S   0.0  0.0  35:51.03 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
        2 root      20   0       0      0      0 S   0.0  0.0   0:01.65 [kthreadd]
        3 root      20   0       0      0      0 S   0.0  0.0   0:06.95 [ksoftirqd/0]
        5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 [kworker/0:0H]
        8 root      rt   0       0      0      0 S   0.0  0.0   0:06.58 [migration/0]
        9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 [rcu_bh]
       10 root      20   0       0      0      0 S   0.0  0.0  43:01.62 [rcu_sched]
    (tf) [ye53nis@node128 drmed-git]$
    
  3. print conda list
    conda list
    
    # packages in environment at /home/ye53nis/.conda/envs/tf:
    #
    # Name                    Version                   Build  Channel
    _libgcc_mutex             0.1                        main
    _openmp_mutex             4.5                       1_gnu
    absl-py                   1.0.0                    pypi_0    pypi
    alembic                   1.7.6                    pypi_0    pypi
    anyio                     2.2.0            py39h06a4308_1
    argon2-cffi               20.1.0           py39h27cfd23_1
    asteval                   0.9.26                   pypi_0    pypi
    astunparse                1.6.3                    pypi_0    pypi
    async_generator           1.10               pyhd3eb1b0_0
    attrs                     21.4.0             pyhd3eb1b0_0
    babel                     2.9.1              pyhd3eb1b0_0
    backcall                  0.2.0              pyhd3eb1b0_0
    bleach                    4.1.0              pyhd3eb1b0_0
    brotlipy                  0.7.0           py39h27cfd23_1003
    ca-certificates           2021.10.26           h06a4308_2
    cachetools                5.0.0                    pypi_0    pypi
    certifi                   2021.10.8        py39h06a4308_2
    cffi                      1.15.0           py39hd667e15_1
    charset-normalizer        2.0.4              pyhd3eb1b0_0
    click                     8.0.3                    pypi_0    pypi
    cloudpickle               2.0.0                    pypi_0    pypi
    cryptography              36.0.0           py39h9ce1e76_0
    cycler                    0.11.0                   pypi_0    pypi
    cython                    0.29.27                  pypi_0    pypi
    databricks-cli            0.16.4                   pypi_0    pypi
    debugpy                   1.5.1            py39h295c915_0
    decorator                 5.1.1              pyhd3eb1b0_0
    defusedxml                0.7.1              pyhd3eb1b0_0
    docker                    5.0.3                    pypi_0    pypi
    entrypoints               0.3              py39h06a4308_0
    fcsfiles                  2022.2.2                 pypi_0    pypi
    flask                     2.0.2                    pypi_0    pypi
    flatbuffers               2.0                      pypi_0    pypi
    fonttools                 4.29.1                   pypi_0    pypi
    future                    0.18.2                   pypi_0    pypi
    gast                      0.5.3                    pypi_0    pypi
    gitdb                     4.0.9                    pypi_0    pypi
    gitpython                 3.1.26                   pypi_0    pypi
    google-auth               2.6.0                    pypi_0    pypi
    google-auth-oauthlib      0.4.6                    pypi_0    pypi
    google-pasta              0.2.0                    pypi_0    pypi
    greenlet                  1.1.2                    pypi_0    pypi
    grpcio                    1.43.0                   pypi_0    pypi
    gunicorn                  20.1.0                   pypi_0    pypi
    h5py                      3.6.0                    pypi_0    pypi
    idna                      3.3                pyhd3eb1b0_0
    importlib-metadata        4.8.2            py39h06a4308_0
    importlib_metadata        4.8.2                hd3eb1b0_0
    ipykernel                 6.4.1            py39h06a4308_1
    ipython                   7.31.1           py39h06a4308_0
    ipython_genutils          0.2.0              pyhd3eb1b0_1
    itsdangerous              2.0.1                    pypi_0    pypi
    jedi                      0.18.1           py39h06a4308_1
    jinja2                    3.0.2              pyhd3eb1b0_0
    joblib                    1.1.0                    pypi_0    pypi
    json5                     0.9.6              pyhd3eb1b0_0
    jsonschema                3.2.0              pyhd3eb1b0_2
    jupyter_client            7.1.2              pyhd3eb1b0_0
    jupyter_core              4.9.1            py39h06a4308_0
    jupyter_server            1.4.1            py39h06a4308_0
    jupyterlab                3.2.1              pyhd3eb1b0_1
    jupyterlab_pygments       0.1.2                      py_0
    jupyterlab_server         2.10.2             pyhd3eb1b0_1
    keras                     2.8.0                    pypi_0    pypi
    keras-preprocessing       1.1.2                    pypi_0    pypi
    kiwisolver                1.3.2                    pypi_0    pypi
    ld_impl_linux-64          2.35.1               h7274673_9
    libclang                  13.0.0                   pypi_0    pypi
    libffi                    3.3                  he6710b0_2
    libgcc-ng                 9.3.0               h5101ec6_17
    libgomp                   9.3.0               h5101ec6_17
    libsodium                 1.0.18               h7b6447c_0
    libstdcxx-ng              9.3.0               hd4cf53a_17
    lmfit                     1.0.3                    pypi_0    pypi
    mako                      1.1.6                    pypi_0    pypi
    markdown                  3.3.6                    pypi_0    pypi
    markupsafe                2.0.1            py39h27cfd23_0
    matplotlib                3.5.1                    pypi_0    pypi
    matplotlib-inline         0.1.2              pyhd3eb1b0_2
    mistune                   0.8.4           py39h27cfd23_1000
    mlflow                    1.23.1                   pypi_0    pypi
    multipletau               0.3.3                    pypi_0    pypi
    nbclassic                 0.2.6              pyhd3eb1b0_0
    nbclient                  0.5.3              pyhd3eb1b0_0
    nbconvert                 6.3.0            py39h06a4308_0
    nbformat                  5.1.3              pyhd3eb1b0_0
    ncurses                   6.3                  h7f8727e_2
    nest-asyncio              1.5.1              pyhd3eb1b0_0
    notebook                  6.4.6            py39h06a4308_0
    numpy                     1.22.2                   pypi_0    pypi
    oauthlib                  3.2.0                    pypi_0    pypi
    openssl                   1.1.1m               h7f8727e_0
    opt-einsum                3.3.0                    pypi_0    pypi
    packaging                 21.3               pyhd3eb1b0_0
    pandas                    1.4.0                    pypi_0    pypi
    pandocfilters             1.5.0              pyhd3eb1b0_0
    parso                     0.8.3              pyhd3eb1b0_0
    pexpect                   4.8.0              pyhd3eb1b0_3
    pickleshare               0.7.5           pyhd3eb1b0_1003
    pillow                    9.0.1                    pypi_0    pypi
    pip                       21.2.4           py39h06a4308_0
    prometheus-flask-exporter 0.18.7                   pypi_0    pypi
    prometheus_client         0.13.1             pyhd3eb1b0_0
    prompt-toolkit            3.0.20             pyhd3eb1b0_0
    protobuf                  3.19.4                   pypi_0    pypi
    ptyprocess                0.7.0              pyhd3eb1b0_2
    pyasn1                    0.4.8                    pypi_0    pypi
    pyasn1-modules            0.2.8                    pypi_0    pypi
    pycparser                 2.21               pyhd3eb1b0_0
    pygments                  2.11.2             pyhd3eb1b0_0
    pyopenssl                 22.0.0             pyhd3eb1b0_0
    pyparsing                 3.0.4              pyhd3eb1b0_0
    pyrsistent                0.18.0           py39heee7806_0
    pysocks                   1.7.1            py39h06a4308_0
    python                    3.9.7                h12debd9_1
    python-dateutil           2.8.2              pyhd3eb1b0_0
    pytz                      2021.3             pyhd3eb1b0_0
    pyyaml                    6.0                      pypi_0    pypi
    pyzmq                     22.3.0           py39h295c915_2
    querystring-parser        1.2.4                    pypi_0    pypi
    readline                  8.1.2                h7f8727e_1
    requests                  2.27.1             pyhd3eb1b0_0
    requests-oauthlib         1.3.1                    pypi_0    pypi
    rsa                       4.8                      pypi_0    pypi
    scikit-learn              1.0.2                    pypi_0    pypi
    scipy                     1.8.0                    pypi_0    pypi
    seaborn                   0.11.2                   pypi_0    pypi
    send2trash                1.8.0              pyhd3eb1b0_1
    setuptools                58.0.4           py39h06a4308_0
    six                       1.16.0             pyhd3eb1b0_0
    smmap                     5.0.0                    pypi_0    pypi
    sniffio                   1.2.0            py39h06a4308_1
    sqlalchemy                1.4.31                   pypi_0    pypi
    sqlite                    3.37.2               hc218d9a_0
    sqlparse                  0.4.2                    pypi_0    pypi
    tabulate                  0.8.9                    pypi_0    pypi
    tensorboard               2.8.0                    pypi_0    pypi
    tensorboard-data-server   0.6.1                    pypi_0    pypi
    tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
    tensorflow                2.8.0                    pypi_0    pypi
    tensorflow-io-gcs-filesystem 0.24.0                   pypi_0    pypi
    termcolor                 1.1.0                    pypi_0    pypi
    terminado                 0.9.4            py39h06a4308_0
    testpath                  0.5.0              pyhd3eb1b0_0
    tf-estimator-nightly      2.8.0.dev2021122109          pypi_0    pypi
    threadpoolctl             3.1.0                    pypi_0    pypi
    tk                        8.6.11               h1ccaba5_0
    tornado                   6.1              py39h27cfd23_0
    traitlets                 5.1.1              pyhd3eb1b0_0
    typing-extensions         4.0.1                    pypi_0    pypi
    tzdata                    2021e                hda174b7_0
    uncertainties             3.1.6                    pypi_0    pypi
    urllib3                   1.26.8             pyhd3eb1b0_0
    wcwidth                   0.2.5              pyhd3eb1b0_0
    webencodings              0.5.1            py39h06a4308_1
    websocket-client          1.2.3                    pypi_0    pypi
    werkzeug                  2.0.3                    pypi_0    pypi
    wheel                     0.37.1             pyhd3eb1b0_0
    wrapt                     1.13.3                   pypi_0    pypi
    xz                        5.2.5                h7b6447c_0
    zeromq                    4.3.4                h2531618_0
    zipp                      3.7.0              pyhd3eb1b0_0
    zlib                      1.2.11               h7f8727e_4
    (tf) [ye53nis@node128 drmed-git]$
    
  4. Show tree of input files used.
    tree ../saves/firstartifact_Nov2020_train_max2sets
    echo --------------------
    tree ../saves/firstartifact_Nov2020_val_max2sets_SORTEDIN
    echo --------------------
    tree ../saves/firstartifact_Nov2020_test
    
    
    (tf) [ye53nis@node128 drmed-git]$ tree ../saves/firstartifact_Nov2020_train_max2sets
    ../saves/firstartifact_Nov2020_train_max2sets
    ├── 0.069
    │   ├── 0.01
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.069_set002.csv
    │   │   └── traces_brightclust_Nov2020_D0.069_set003.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.069_set009.csv
    ├── 0.08
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.08_set007.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.08_set002.csv
    │   │   └── traces_brightclust_Nov2020_D0.08_set006.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.08_set004.csv
    │       └── traces_brightclust_Nov2020_D0.08_set009.csv
    ├── 0.1
    │   ├── 0.01
    │   │   ├── traces_brightclust_Nov2020_D0.1_set004.csv
    │   │   └── traces_brightclust_Nov2020_D0.1_set006.csv
    │   ├── 0.1
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.1_set003.csv
    │       └── traces_brightclust_Nov2020_D0.1_set007.csv
    ├── 0.2
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.2_set003.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.2_set001.csv
    │   │   └── traces_brightclust_Nov2020_D0.2_set004.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.2_set009.csv
    │       └── traces_brightclust_Nov2020_D0.2_set010.csv
    ├── 0.4
    │   ├── 0.01
    │   │   ├── traces_brightclust_Nov2020_D0.4_set004.csv
    │   │   └── traces_brightclust_Nov2020_D0.4_set010.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.4_set002.csv
    │   │   └── traces_brightclust_Nov2020_D0.4_set003.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.4_set006.csv
    │       └── traces_brightclust_Nov2020_D0.4_set007.csv
    ├── 0.6
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.6_set010.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D0.6_set004.csv
    │   │   └── traces_brightclust_Nov2020_D0.6_set005.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D0.6_set001.csv
    │       └── traces_brightclust_Nov2020_D0.6_set002.csv
    ├── 10
    │   ├── 0.01
    │   │   ├── traces_brightclust_Nov2020_D10_set003.csv
    │   │   └── traces_brightclust_Nov2020_D10_set004.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D10_set006.csv
    │   │   └── traces_brightclust_Nov2020_D10_set007.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D10_set010.csv
    ├── 1.0
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D1.0_set010.csv
    │   ├── 0.1
    │   │   ├── traces_brightclust_Nov2020_D1.0_set004.csv
    │   │   └── traces_brightclust_Nov2020_D1.0_set007.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D1.0_set001.csv
    │       └── traces_brightclust_Nov2020_D1.0_set002.csv
    ├── 3.0
    │   ├── 0.01
    │   │   ├── traces_brightclust_Nov2020_D3.0_set005.csv
    │   │   └── traces_brightclust_Nov2020_D3.0_set006.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D3.0_set010.csv
    │   └── 1.0
    │       ├── traces_brightclust_Nov2020_D3.0_set001.csv
    │       └── traces_brightclust_Nov2020_D3.0_set003.csv
    └── 50
        ├── 0.01
        │   └── traces_brightclust_Nov2020_D50_set006.csv
        ├── 0.1
        │   ├── traces_brightclust_Nov2020_D50_set009.csv
        │   └── traces_brightclust_Nov2020_D50_set010.csv
        └── 1.0
            ├── traces_brightclust_Nov2020_D50_set004.csv
            └── traces_brightclust_Nov2020_D50_set005.csv
    
    40 directories, 48 files
    (tf) [ye53nis@node128 drmed-git]$ echo --------------------
    --------------------
    (tf) [ye53nis@node128 drmed-git]$ tree ../saves/firstartifact_Nov2020_val_max2sets_SORTEDIN
    ../saves/firstartifact_Nov2020_val_max2sets_SORTEDIN
    ├── 0.069
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.069_set006.csv
    │   └── 1.0
    ├── 0.08
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.08_set008.csv
    │   └── 1.0
    ├── 0.1
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.1_set008.csv
    │   ├── 0.1
    │   └── 1.0
    ├── 0.2
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.2_set006.csv
    │   └── 1.0
    ├── 0.4
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.4_set009.csv
    │   └── 1.0
    ├── 0.6
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.6_set006.csv
    │   └── 1.0
    ├── 10
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D10_set008.csv
    │   ├── 0.1
    │   └── 1.0
    ├── 1.0
    │   ├── 0.01
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D1.0_set009.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D1.0_set008.csv
    ├── 3.0
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D3.0_set008.csv
    │   ├── 0.1
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D3.0_set009.csv
    └── 50
        ├── 0.01
        ├── 0.1
        └── 1.0
            └── traces_brightclust_Nov2020_D50_set007.csv
    
    40 directories, 12 files
    (tf) [ye53nis@node128 drmed-git]$ echo --------------------
    --------------------
    
    (tf) [ye53nis@node128 drmed-git]$ tree ../saves/firstartifact_Nov2020_test
    ../saves/firstartifact_Nov2020_test
    ├── 0.069
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.069_set005.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.069_set001.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.069_set010.csv
    ├── 0.08
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.08_set005.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.08_set003.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.08_set001.csv
    ├── 0.1
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.1_set002.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.1_set005.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.1_set001.csv
    ├── 0.2
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.2_set002.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.2_set007.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.2_set005.csv
    ├── 0.4
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.4_set008.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.4_set001.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.4_set005.csv
    ├── 0.6
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D0.6_set008.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D0.6_set003.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D0.6_set009.csv
    ├── 10
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D10_set002.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D10_set001.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D10_set005.csv
    ├── 1.0
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D1.0_set006.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D1.0_set003.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D1.0_set005.csv
    ├── 3.0
    │   ├── 0.01
    │   │   └── traces_brightclust_Nov2020_D3.0_set004.csv
    │   ├── 0.1
    │   │   └── traces_brightclust_Nov2020_D3.0_set007.csv
    │   └── 1.0
    │       └── traces_brightclust_Nov2020_D3.0_set002.csv
    └── 50
        ├── 0.01
        │   └── traces_brightclust_Nov2020_D50_set002.csv
        ├── 0.1
        │   └── traces_brightclust_Nov2020_D50_set003.csv
        └── 1.0
            └── traces_brightclust_Nov2020_D50_set001.csv
    
    40 directories, 30 files
    (tf) [ye53nis@node128 drmed-git]$
    

2.6.6 Setup: Show all hyperparameters that worked in exp-210807-hparams

  1. get the comparison data of all runs from exp-210807-hparams via git restore
    git show 04e9dc3:./data/exp-210807-hparams/mlflow/run1-2_comparison.csv > ./data/exp-220227-unet/mlflow/exp-210807-hparams_comparison.csv
    
  2. open the file in jupyter, do some processing to only find the best runs, and display the relevant hparams (see exp-210807-hparams section 4. Analyze run 1 and 2 for explanations on the processing)
    %cd /beegfs/ye53nis/drmed-git
    import numpy as np
    import pandas as pd
    
    /beegfs/ye53nis/drmed-git
    
    run1_2 = pd.read_csv('data/exp-220227-unet/mlflow/exp-210807-hparams_comparison.csv', index_col=0)
    run1_2_valauc = run1_2.loc['val_auc'].astype(float)
    singles_ls = ['5441e71efe0f4dae868648e7cc795c65']
    
    run1_2_singles = run1_2.loc[:, singles_ls]
    run1_2_singles.iloc[35:, :] = run1_2_singles.iloc[35:, :].astype(np.float64)
    run1_2 = run1_2.drop(columns=singles_ls)
    
    assert len(run1_2.iloc[35:, :].columns) % 2 == 0
    
    run1_2_doubleparams = pd.DataFrame()
    run1_2_doublemetrics = pd.DataFrame()
    double_cols = []
    for left, right in zip(run1_2.iloc[:, ::2].items(), run1_2.iloc[:, 1::2].items()):
        double_cols.append((left[0], right[0]))
        current_metrics = left[1].iloc[35:].combine(other=right[1].iloc[35:],
                                                    func=(lambda x1, x2: (float(x1) + float(x2)) / 2))
        current_params = left[1].iloc[:35].combine(other=right[1].iloc[:35],
                                                   func=(lambda x1, x2: set((x1, x2)) if x1 != x2 else x1))
        run1_2_doubleparams = pd.concat([run1_2_doubleparams, current_params], axis=1)
        run1_2_doublemetrics = pd.concat([run1_2_doublemetrics, current_metrics], axis=1)
    
    run1_2_doublemetrics = pd.DataFrame(data=run1_2_doublemetrics.to_numpy(),
                                        index=run1_2.iloc[35:, :].index,
                                        columns=double_cols)
    
    run1_2_doubleparams = pd.DataFrame(data=run1_2_doubleparams.to_numpy(),
                                       index=run1_2.iloc[:35, :].index,
                                       columns=double_cols)
    
    run1_2_combimetrics = pd.concat([run1_2_doublemetrics, run1_2_singles.iloc[35:, :]], axis=1)
    run1_2_combiparams = pd.concat([run1_2_doubleparams, run1_2_singles.iloc[:35, :]], axis=1)
    run1_2_mymetrics = run1_2_combimetrics.loc[['val_auc', 'val_recall0.5', 'val_precision0.5']]
    run1_2_myparams = run1_2_combiparams.loc[['hp_batch_size', 'hp_first_filters', 'hp_input_size', 'hp_lr_power', 'hp_lr_start', 'hp_n_levels', 'hp_pool_size', 'hp_scaler']]
    run1_2_my = pd.concat([run1_2_mymetrics, run1_2_myparams], axis=0).T
    # cond1 = run1_2_combimetrics.loc[:, 'val_auc'] > 0.95
    cond2 = run1_2_my.loc[:, 'val_recall0.5'] > 0.85
    cond3 = run1_2_my.loc[:, 'val_precision0.5'] > 0.85
    
    with pd.option_context('display.max_rows', None, 'display.max_columns', None):
        display(run1_2_my.loc[cond2 & cond3])
    
    Run ID: valauc valrecall0.5 valprecision0.5 hpbatchsize hpfirstfilters hpinputsize hplrpower hplrstart hpnlevels hppoolsize hpscaler
    (9051e32b87d84f3485b980067addec30, 61ff87bdb89b4e2ba64f8dacc774992d) 0.981 0.8975 0.918 26 44 16384 1 0.0136170138242663 7 2 standard
    (93b168c0ff7942c8a908a94129daf973, f243b3b742de4dbcb7ccfbd4244706f8) 0.976 0.893 0.852 15 23 16384 7 0.0305060808685107 6 4 quantg
    (a5b8551144ff46e697a39cd1551e1475, 98cf8cdef9c54b5286e277e75e2ab8c1) 0.984 0.916 0.909 20 78 16384 4 0.0584071108418767 4 4 standard
    (00f2635d9fa2463c9a066722163405be, d0a8e1748b194f3290d471b6b44f19f8) 0.987 0.929 0.9065 28 6 16384 1 0.0553313915596308 5 4 minmax
    (5604d43c1ece461b8e6eaa0dfb65d6dc, 3612536a77f34f22bc83d1d809140aa6) 0.9745 0.885 0.8985 20 128 16384 1 0.043549707353273 3 4 standard
    (7cafab027cdd4fc9bf20a43e989df510, 16dff15d935f45e2a836b1f41b07b4e3) 0.978 0.8905 0.891 10 16 8192 1 0.0627676336651573 5 4 robust
    (0e328920e86049928202db95e8cfb7be, bf9d2725eb16462d9a101f0a077ce2b5) 0.976 0.875 0.888 14 16 16384 5 0.0192390310290551 9 2 robust
    (1c954fbc02b747bc813c587ac703c74a, ba49a80c2616407a8f1fe1fd12096fe0) 0.962 0.856 0.8585 17 16 16384 5 0.0101590069352232 3 4 l2
    (3cbd945b62ec4634839372e403f6f377, 458b36a70db843719d202a8eda448f17) 0.972 0.872 0.9135 9 64 4096 1 0.0100697459464075 5 4 maxabs
  3. notes on hparams:
    • I used three different input sizes in hparams training (4096, 8192, 16384). As experimental test data I have got traces which are around 8000 and traces which are around 32000 time steps. To balance between both, I will only use 16384 as an input size.
    • The UNET only excepts input sizes which are exactly the power of 2 and > 1024. To deal with that for experimental traces which have a different size, I append the median of the trace until it reaches the next biggest power of 2. That means in the script below my input_size will be 14000, so each trace will be padded with 2384 median values (~15% of the input), and the corresponding labels will be 0.
    • the epoch size used for training will be 100 epochs for each hparam configuration

2.6.7 Setup: Fitting

  • all fitting was done in Dominic Waithe's focuspoint. (see https://pubmed.ncbi.nlm.nih.gov/26589275/ and https://github.com/dwaithe/FCS_point_correlator)
  • I installed the program by cloning the github and following the setup instructions there. Some packages were missing to run the program, the following is the conda environment with focuspoint and all needed packages
    conda list -n focus
    
    # packages in environment at /home/lex/Programme/miniconda3/envs/focus:
    #          
    # Name Version Build Channel  
    _libgccmutex 0.1 main      
    _openmpmutex 4.5 1gnu      
    asteval 0.9.26 pypi0 pypi    
    blas 1.0 mkl      
    ca-certificates 2022.3.18 h06a43080      
    certifi 2021.10.8 py39h06a43082      
    cycler 0.11.0 pypi0 pypi    
    cython 0.29.28 pypi0 pypi    
    dbus 1.13.18 hb2f20db0      
    expat 2.4.4 h295c9150      
    focuspoint 0.1 pypi0 pypi    
    fontconfig 2.13.1 h6c099310      
    fonttools 4.31.2 pypi0 pypi    
    freetype 2.11.0 h70c03450      
    future 0.18.2 pypi0 pypi    
    glib 2.69.1 h4ff587b1      
    gst-plugins-base 1.14.0 h8213a912      
    gstreamer 1.14.0 h28cd5cc2      
    icu 58.2 he6710b03      
    intel-openmp 2021.4.0 h06a43083561      
    jpeg 9d h7f8727e0      
    kiwisolver 1.4.0 pypi0 pypi    
    ldimpllinux-64 2.35.1 h72746739      
    libffi 3.3 he6710b02      
    libgcc-ng 9.3.0 h5101ec617      
    libgomp 9.3.0 h5101ec617      
    libpng 1.6.37 hbc830470      
    libstdcxx-ng 9.3.0 hd4cf53a17      
    libuuid 1.0.3 h7f8727e2      
    libxcb 1.14 h7b6447c0      
    libxml2 2.9.12 h03d6c580      
    lmfit 1.0.3 pypi0 pypi    
    matplotlib 3.5.1 pypi0 pypi    
    mkl 2021.4.0 h06a4308640      
    mkl-service 2.4.0 py39h7f8727e0      
    mklfft 1.3.1 py39hd3c417c0      
    mklrandom 1.2.2 py39h51133e40      
    ncurses 6.3 h7f8727e2      
    numexpr 2.8.1 pypi0 pypi    
    numpy 1.21.2 py39h20f2e390      
    numpy-base 1.21.2 py39h79a11010      
    openssl 1.1.1n h7f8727e0      
    packaging 21.3 pypi0 pypi    
    pcre 8.45 h295c9150      
    pillow 9.0.1 pypi0 pypi    
    pip 21.2.4 py39h06a43080      
    pyparsing 3.0.7 pypi0 pypi    
    pyperclip 1.8.2 pypi0 pypi    
    pyqt 5.9.2 py39h25316186      
    python 3.9.7 h12debd91      
    python-dateutil 2.8.2 pypi0 pypi    
    qt 5.9.7 h5867ecd1      
    readline 8.1.2 h7f8727e1      
    scipy 1.8.0 pypi0 pypi    
    setuptools 58.0.4 py39h06a43080      
    sip 4.19.13 py39h295c9150      
    six 1.16.0 pyhd3eb1b01      
    sqlite 3.38.0 hc218d9a0      
    tables 3.7.0 pypi0 pypi    
    tifffile 2022.3.16 pypi0 pypi    
    tk 8.6.11 h1ccaba50      
    tzdata 2021e hda174b70      
    uncertainties 3.1.6 pypi0 pypi    
    wheel 0.37.1 pyhd3eb1b00      
    xz 5.2.5 h7b6447c0      
    zlib 1.2.11 h7f8727e4      
  • this is how the program is started
    conda activate focus
    python -m focuspoint.FCS_point_correlator
    
  • then:
    1. go to the 'Fit Function' tab
    2. load correlated files, e.g. from ./data/exp-220227-unet/2022-05-22_experimental-af488/clean
    3. inside the base directory of the experiment (e.g. ./data/exp-220227-unet/2022-05-22_experimental-af488) I saved the focuspoint profile file, which you can load to reproduce all my fit settings.
      1. for 1-component fit: ./data/exp-220227-unet/2022-05-22_experimental-af488/af488+luvs_1comp.profile
      2. for 2-component fit: ./data/exp-220227-unet/2022-05-22_experimental-af488/af488+luvs_2comp.profile
    4. the setting which is not saved in the .profile file is the fit range (xmin, xmax). This is saved in all the ...outParam.csv files. Generally, this was
      • for af488 data:
        • xmin=0.001
        • for averaging correction: xmax=0.5
        • for data without peak artifacts (only af488) and all other correction methods: xmax=100
        • for data with peak artifacts (only af488+luvs) and all other correction methods: xmax=500
      • for pex5 data:
        • xmin=0.001
        • for averaging correction: xmax=0.5
        • for all other correction methods: xmax=1000
      • simulated data:
        • xmin=1
        • xmax=8000 - note: sometimes the correction methods did shorten the traces below 8000ms. Then, the maximum correlation length was automatically shortened by the program. In exp-220227-unet, I did not plot the comparison of fit outcomes of simulated data, so for details on these xmax values see the assert statements when loading simulated data in exp-220316-publication1

2.6.8 Experiment 1: run training of 9 promising hparam combinations

  1. run training of (9051e32b87d84f3485b980067addec30, 61ff87bdb89b4e2ba64f8dacc774992d) with hparams from above.
    mlflow run . -e main -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ \
           -P batch_size=26 \
           -P first_filters=44 \
           -P input_size=14000 \
           -P lr_start=0.0136170138242663 \
           -P lr_power=1 \
           -P epochs=100 \
           -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets \
           -P csv_path_val=/beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN \
           -P scaler=standard \
           -P n_levels=7 \
           -P pool_size=2
    
    INFO: 'exp-220227-unet' does not exist. Creating a new experiment
    2022/02/27 23:06:17 INFO mlflow.projects.utils: === Created directory /tmp/tmp5u38tprq for downloading remote URIs passed to arguments of type 'path' ===
    2022/02/27 23:06:17 INFO mlflow.projects.backend.local: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe 1>&2 && python src/fluotracify/train
    ing/train.py --fluotracify_path /beegfs/ye53nis/drmed-git/src --batch_size 26 --input_size 14000 --lr_start 0.0136170138242663 --lr_power 1 --epochs 100 --csv_path_train /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets --csv_p
    ath_val /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN --col_per_example 3 --scaler standard --n_levels 7 --first_filters 44 --pool_size 2' in run with ID '484af471c61943fa90e5f78e78a229f0' ===
    2022-02-27 23:06:19.459522: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-02-27 23:06:27,328 - train -  Python version: 3.9.6 (default, Jul 30 2021, 16:35:19)
    [GCC 7.5.0]
    2022-02-27 23:06:27,329 - train -  Tensorflow version: 2.5.0
    2022-02-27 23:06:27,329 - train -  tf.keras version: 2.5.0
    2022-02-27 23:06:27,329 - train -  Cudnn version: 8
    2022-02-27 23:06:27,329 - train -  Cuda version: 11.2
    2022-02-27 23:06:27.332616: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
    2022-02-27 23:06:27.373032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-02-27 23:06:27.373166: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-02-27 23:06:27.382894: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-02-27 23:06:27.382990: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    2022-02-27 23:06:27.386881: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
    2022-02-27 23:06:27.389946: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
    2022-02-27 23:06:27.399178: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
    2022-02-27 23:06:27.413770: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
    2022-02-27 23:06:27.415902: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-02-27 23:06:27.419520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-02-27 23:06:27,419 - train -  GPUs: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]. Trying to set memory growth to "True"...
    2022-02-27 23:06:27,420 - train -  Setting memory growth successful.
    2022-02-27 23:06:33,649 - train -  1/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set004.csv
    2022-02-27 23:06:47,474 - train -  2/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set002.csv
    2022-02-27 23:06:52,553 - train -  3/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set001.csv
    2022-02-27 23:06:56,374 - train -  4/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set007.csv
    2022-02-27 23:06:59,968 - train -  5/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
    2022-02-27 23:07:03,508 - train -  6/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/1.0/traces_brightclust_Nov2020_D10_set010.csv
    2022-02-27 23:07:07,092 - train -  7/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set004.csv
    2022-02-27 23:07:10,718 - train -  8/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.1/traces_brightclust_Nov2020_D3.0_set010.csv
    2022-02-27 23:07:14,433 - train -  9/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set009.csv
    2022-02-27 23:07:18,054 - train -  10/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set003.csv
    2022-02-27 23:07:21,398 - train -  11/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set007.csv
    2022-02-27 23:07:33,241 - train -  12/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
    2022-02-27 23:07:37,082 - train -  13/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set003.csv
    2022-02-27 23:07:40,911 - train -  14/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set002.csv
    2022-02-27 23:07:44,813 - train -  15/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set005.csv
    2022-02-27 23:07:48,436 - train -  16/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/1.0/traces_brightclust_Nov2020_D0.069_set009.csv
    2022-02-27 23:07:52,998 - train -  17/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set004.csv
    2022-02-27 23:07:56,536 - train -  18/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set010.csv
    2022-02-27 23:08:00,088 - train -  19/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set009.csv
    2022-02-27 23:08:03,661 - train -  20/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.01/traces_brightclust_Nov2020_D1.0_set010.csv
    2022-02-27 23:08:07,253 - train -  21/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set007.csv
    2022-02-27 23:08:22,306 - train -  22/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set010.csv
    2022-02-27 23:08:25,939 - train -  23/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set001.csv
    2022-02-27 23:08:29,797 - train -  24/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set003.csv
    2022-02-27 23:08:33,419 - train -  25/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set003.csv
    2022-02-27 23:08:38,026 - train -  26/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set004.csv
    2022-02-27 23:08:41,552 - train -  27/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set001.csv
    2022-02-27 23:08:46,334 - train -  28/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set005.csv
    2022-02-27 23:08:52,449 - train -  29/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set006.csv
    2022-02-27 23:09:00,390 - train -  30/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set004.csv
    2022-02-27 23:09:13,802 - train -  31/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set006.csv
    2022-02-27 23:09:17,690 - train -  32/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set004.csv
    2022-02-27 23:09:21,264 - train -  33/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set005.csv
    2022-02-27 23:09:25,460 - train -  34/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set003.csv
    2022-02-27 23:09:29,041 - train -  35/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.01/traces_brightclust_Nov2020_D0.2_set003.csv
    2022-02-27 23:09:33,185 - train -  36/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.01/traces_brightclust_Nov2020_D50_set006.csv
    2022-02-27 23:09:36,927 - train -  37/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set004.csv
    2022-02-27 23:09:43,792 - train -  38/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set004.csv
    2022-02-27 23:09:55,806 - train -  39/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.01/traces_brightclust_Nov2020_D0.6_set010.csv
    2022-02-27 23:09:59,340 - train -  40/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set007.csv
    2022-02-27 23:10:04,195 - train -  41/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set006.csv
    2022-02-27 23:10:07,769 - train -  42/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set002.csv
    2022-02-27 23:10:11,494 - train -  43/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set006.csv
    2022-02-27 23:10:15,480 - train -  44/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set006.csv
    2022-02-27 23:10:19,534 - train -  45/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set010.csv
    2022-02-27 23:10:23,308 - train -  46/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.01/traces_brightclust_Nov2020_D0.08_set007.csv
    2022-02-27 23:10:27,088 - train -  47/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set002.csv
    2022-02-27 23:10:30,870 - train -  48/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set009.csv
    2022-02-27 23:10:34,365 - train -  1/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/0.1/traces_brightclust_Nov2020_D1.0_set009.csv
    2022-02-27 23:10:43,346 - train -  2/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/50/1.0/traces_brightclust_Nov2020_D50_set007.csv
    2022-02-27 23:10:59,488 - train -  3/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.4/0.1/traces_brightclust_Nov2020_D0.4_set009.csv
    2022-02-27 23:11:07,159 - train -  4/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/1.0/traces_brightclust_Nov2020_D3.0_set009.csv
    2022-02-27 23:11:11,824 - train -  5/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.1/0.01/traces_brightclust_Nov2020_D0.1_set008.csv
    2022-02-27 23:11:15,330 - train -  6/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/10/0.01/traces_brightclust_Nov2020_D10_set008.csv
    2022-02-27 23:11:19,613 - train -  7/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.08/0.1/traces_brightclust_Nov2020_D0.08_set008.csv
    2022-02-27 23:11:22,985 - train -  8/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
    2022-02-27 23:11:26,571 - train -  9/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/0.01/traces_brightclust_Nov2020_D3.0_set008.csv
    2022-02-27 23:11:30,142 - train -  10/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.2/0.1/traces_brightclust_Nov2020_D0.2_set006.csv
    2022-02-27 23:11:33,712 - train -  11/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.069/0.1/traces_brightclust_Nov2020_D0.069_set006.csv
    2022-02-27 23:11:40,154 - train -  12/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.6/0.1/traces_brightclust_Nov2020_D0.6_set006.csv
    2022-02-27 23:11:40,374 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 4800), (16384, 4800), (16384, 4800)]
    2022-02-27 23:11:40,463 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 1200), (16384, 1200), (16384, 1200)]
    2022-02-27 23:11:40.616724: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operatio
    ns:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    2022-02-27 23:11:40.620393: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-02-27 23:11:40.624428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-02-27 23:11:40.624579: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-02-27 23:11:41.105011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
    2022-02-27 23:11:41.105083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0
    2022-02-27 23:11:41.105097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N
    2022-02-27 23:11:41.107958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15403 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, p
    ci bus id: 0000:82:00.0, compute capability: 6.0)
    2022-02-27 23:11:43,160 - train -  number of examples: 4800
    2022-02-27 23:11:43,526 - train -  number of examples: 1200
    2022-02-27 23:11:46,063 - train -  unet: input shape: (None, None, 1), output shape: (None, None, 1)
    2022/02/27 23:11:46 WARNING mlflow.utils.autologging_utils: Encountered unexpected error during tensorflow autologging: Changing param values is not allowed. Param with key='batch_size' was already logged with value='26' for run ID='484
    af471c61943fa90e5f78e78a229f0'. Attempted logging new value 'None'.
    Epoch 1/100
    2022-02-27 23:11:56.021789: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
    2022-02-27 23:11:56.201257: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199895000 Hz
    2022-02-27 23:12:06.010819: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-02-27 23:12:06.325533: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101
    2022-02-27 23:12:06.806608: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-02-27 23:12:07.089703: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    184/184 [==============================] - 212s 1s/step - loss: 1.1422 - tp0.1: 8390499.0000 - fp0.1: 12828878.0000 - tn0.1: 55341968.0000 - fn0.1: 1819714.0000 - precision0.1: 0.3954 - recall0.1: 0.8218 - tp0.3: 7189476.0000 - fp0.3: 4
    881676.0000 - tn0.3: 63289180.0000 - fn0.3: 3020737.0000 - precision0.3: 0.5956 - recall0.3: 0.7041 - tp0.5: 6041120.0000 - fp0.5: 2184567.0000 - tn0.5: 65986268.0000 - fn0.5: 4169093.0000 - precision0.5: 0.7344 - recall0.5: 0.5917 - tp
    0.7: 4490541.0000 - fp0.7: 712795.0000 - tn0.7: 67458040.0000 - fn0.7: 5719672.0000 - precision0.7: 0.8630 - recall0.7: 0.4398 - tp0.9: 2966847.0000 - fp0.9: 153266.0000 - tn0.9: 68017552.0000 - fn0.9: 7243366.0000 - precision0.9: 0.950
    9 - recall0.9: 0.2906 - accuracy: 0.9189 - auc: 0.8890 - f1: 0.6554 - val_loss: 45.6530 - val_tp0.1: 2695397.0000 - val_fp0.1: 16186602.0000 - val_tn0.1: 695010.0000 - val_fn0.1: 18255.0000 - val_precision0.1: 0.1427 - val_recall0.1: 0.
    9933 - val_tp0.3: 2693432.0000 - val_fp0.3: 16009674.0000 - val_tn0.3: 871938.0000 - val_fn0.3: 20220.0000 - val_precision0.3: 0.1440 - val_recall0.3: 0.9925 - val_tp0.5: 2691511.0000 - val_fp0.5: 15860014.0000 - val_tn0.5: 1021598.0000
     - val_fn0.5: 22141.0000 - val_precision0.5: 0.1451 - val_recall0.5: 0.9918 - val_tp0.7: 2688428.0000 - val_fp0.7: 15650703.0000 - val_tn0.7: 1230909.0000 - val_fn0.7: 25224.0000 - val_precision0.7: 0.1466 - val_recall0.7: 0.9907 - val_
    tp0.9: 2681968.0000 - val_fp0.9: 15334839.0000 - val_tn0.9: 1546773.0000 - val_fn0.9: 31684.0000 - val_precision0.9: 0.1489 - val_recall0.9: 0.9883 - val_accuracy: 0.1895 - val_auc: 0.5616 - val_f1: 0.2531
    
    ...
    
    Epoch 100/100
    184/184 [==============================] - 184s 1s/step - loss: 0.0877 - tp0.1: 10015121.0000 - fp0.1: 1833216.0000 - tn0.1: 66390352.0000 - fn0.1: 142350.0000 - precision0.1: 0.8453 - recall0.1: 0.9860 - tp0.3: 9874038.0000 - fp0.3: 10
    86322.0000 - tn0.3: 67137280.0000 - fn0.3: 283433.0000 - precision0.3: 0.9009 - recall0.3: 0.9721 - tp0.5: 9674727.0000 - fp0.5: 664419.0000 - tn0.5: 67559176.0000 - fn0.5: 482744.0000 - precision0.5: 0.9357 - recall0.5: 0.9525 - tp0.7:
     9343030.0000 - fp0.7: 341635.0000 - tn0.7: 67881952.0000 - fn0.7: 814441.0000 - precision0.7: 0.9647 - recall0.7: 0.9198 - tp0.9: 8626370.0000 - fp0.9: 88068.0000 - tn0.9: 68135504.0000 - fn0.9: 1531101.0000 - precision0.9: 0.9899 - re
    call0.9: 0.8493 - accuracy: 0.9854 - auc: 0.9920 - f1: 0.9440 - val_loss: 0.1556 - val_tp0.1: 2623480.0000 - val_fp0.1: 609139.0000 - val_tn0.1: 16273852.0000 - val_fn0.1: 88793.0000 - val_precision0.1: 0.8116 - val_recall0.1: 0.9673 -
    val_tp0.3: 2573475.0000 - val_fp0.3: 375340.0000 - val_tn0.3: 16507651.0000 - val_fn0.3: 138798.0000 - val_precision0.3: 0.8727 - val_recall0.3: 0.9488 - val_tp0.5: 2518254.0000 - val_fp0.5: 251870.0000 - val_tn0.5: 16631121.0000 - val_
    fn0.5: 194019.0000 - val_precision0.5: 0.9091 - val_recall0.5: 0.9285 - val_tp0.7: 2439448.0000 - val_fp0.7: 153092.0000 - val_tn0.7: 16729899.0000 - val_fn0.7: 272825.0000 - val_precision0.7: 0.9409 - val_recall0.7: 0.8994 - val_tp0.9:
     2279822.0000 - val_fp0.9: 61468.0000 - val_tn0.9: 16821524.0000 - val_fn0.9: 432451.0000 - val_precision0.9: 0.9737 - val_recall0.9: 0.8406 - val_accuracy: 0.9772 - val_auc: 0.9814 - val_f1: 0.9187
    2022-02-28 04:24:39.818514: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
    2022/02/28 04:24:58 INFO mlflow.projects: === Run (ID '484af471c61943fa90e5f78e78a229f0') succeeded ===
    (tf) [ye53nis@node128 drmed-git]$
    
    • name of run: 484af471c61943fa90e5f78e78a229f0
    • metrics after 100th epoch:
      • precisionval,0.5: 0.9091 - recallval,0.5: 0.9285 - f1val,0.5: 0.9187
      • aucval: 0.9814
    • a note on the metrics notation:
      • val means: metrics computed on validation dataset, which was used in training to e.g. ensure no overfitting occurs.
      • 0.5 means: a threshold of 0.5 was applied to the predicted output (floating point values between 0 and 1). That means we get a binary classification - and this classification was compared to the ground truth, which was binary as well. Then, precision, recall, f1, etc were computed
      • the auc function uses 100 different thresholds for computation, so there no own threshold has to be set.
  2. run training of (93b168c0ff7942c8a908a94129daf973, f243b3b742de4dbcb7ccfbd4244706f8) with hparams from above
    mlflow run . -e main -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ \
           -P batch_size=15 \
           -P first_filters=23 \
           -P input_size=14000 \
           -P lr_start=0.0305060808685107 \
           -P lr_power=7 \
           -P epochs=100 \
           -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets \
           -P csv_path_val=/beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN \
           -P scaler=quant_g \
           -P n_levels=6 \
           -P pool_size=4
    
    2022/02/28 14:14:59 INFO mlflow.projects.utils: === Created directory /tmp/tmpjco1jnk_ for downloading remote URIs passed to arguments of type 'path' ===
    2022/02/28 14:15:00 INFO mlflow.projects.backend.local: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe 1>&2 && python src/fluotracify/train
    ing/train.py --fluotracify_path /beegfs/ye53nis/drmed-git/src --batch_size 15 --input_size 14000 --lr_start 0.0305060808685107 --lr_power 7 --epochs 100 --csv_path_train /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets --csv_p
    ath_val /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN --col_per_example 3 --scaler quant_g --n_levels 6 --first_filters 23 --pool_size 4' in run with ID '0cd2023eeaf745aca0d3e8ad5e1fc653' ===
    2022-02-28 14:15:13.580296: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-02-28 14:15:25,693 - train -  Python version: 3.9.6 (default, Jul 30 2021, 16:35:19)
    [GCC 7.5.0]
    2022-02-28 14:15:25,693 - train -  Tensorflow version: 2.5.0
    2022-02-28 14:15:25,693 - train -  tf.keras version: 2.5.0
    2022-02-28 14:15:25,693 - train -  Cudnn version: 8
    2022-02-28 14:15:25,693 - train -  Cuda version: 11.2
    2022-02-28 14:15:25.695839: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
    2022-02-28 14:15:25.776740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-02-28 14:15:25.776884: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-02-28 14:15:25.786987: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-02-28 14:15:25.787115: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    2022-02-28 14:15:25.790205: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
    2022-02-28 14:15:25.791541: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
    2022-02-28 14:15:25.799938: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
    2022-02-28 14:15:25.821517: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
    2022-02-28 14:15:25.822676: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-02-28 14:15:25.826275: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-02-28 14:15:25,826 - train -  GPUs: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]. Trying to set memory growth to "True"...
    2022-02-28 14:15:25,827 - train -  Setting memory growth successful.
    2022-02-28 14:15:33,830 - train -  1/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set004.csv
    2022-02-28 14:15:46,000 - train -  2/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set002.csv
    2022-02-28 14:15:50,090 - train -  3/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set001.csv
    2022-02-28 14:16:02,981 - train -  4/48: /beegfs/ye53nis/saves/first
    artifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set007.csv
    2022-02-28 14:16:11,106 - train -  5/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
    2022-02-28 14:16:16,436 - train -  6/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/1.0/traces_brightclust_Nov2020_D10_set010.csv
    2022-02-28 14:16:21,251 - train -  7/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set004.csv
    2022-02-28 14:16:24,785 - train -  8/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.1/traces_brightclust_Nov2020_D3.0_set010.csv
    2022-02-28 14:16:41,395 - train -  9/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set009.csv
    2022-02-28 14:16:45,170 - train -  10/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set003.csv
    2022-02-28 14:16:48,894 - train -  11/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set007.csv
    2022-02-28 14:17:01,362 - train -  12/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
    2022-02-28 14:17:05,972 - train -  13/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set003.csv
    2022-02-28 14:17:11,392 - train -  14/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set002.csv
    2022-02-28 14:17:15,150 - train -  15/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set005.csv
    2022-02-28 14:17:18,880 - train -  16/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/1.0/traces_brightclust_Nov2020_D0.069_set009.csv
    2022-02-28 14:17:22,858 - train -  17/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set004.csv
    2022-02-28 14:17:26,879 - train -  18/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set010.csv
    2022-02-28 14:17:31,181 - train -  19/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set009.csv
    2022-02-28 14:17:34,897 - train -  20/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.01/traces_brightclust_Nov2020_D1.0_set010.csv
    2022-02-28 14:17:38,846 - train -  21/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set007.csv
    2022-02-28 14:18:01,434 - train -  22/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set010.csv
    2022-02-28 14:18:05,125 - train -  23/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set001.csv
    2022-02-28 14:18:10,548 - train -  24/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set003.csv
    2022-02-28 14:18:15,933 - train -  25/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set003.csv
    2022-02-28 14:18:20,727 - train -  26/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set004.csv
    2022-02-28 14:18:24,064 - train -  27/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set001.csv
    2022-02-28 14:18:35,979 - train -  28/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set005.csv
    2022-02-28 14:18:39,881 - train -  29/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set006.csv
    2022-02-28 14:18:43,394 - train -  30/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set004.csv
    2022-02-28 14:18:53,160 - train -  31/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set006.csv
    2022-02-28 14:18:57,332 - train -  32/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set004.csv
    2022-02-28 14:19:03,503 - train -  33/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set005.csv
    2022-02-28 14:19:08,665 - train -  34/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set003.csv
    2022-02-28 14:19:13,327 - train -  35/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.01/traces_brightclust_Nov2020_D0.2_set003.csv
    2022-02-28 14:19:17,293 - train -  36/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.01/traces_brightclust_Nov2020_D50_set006.csv
    2022-02-28 14:19:21,119 - train -  37/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set004.csv
    2022-02-28 14:19:24,804 - train -  38/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set004.csv
    2022-02-28 14:19:31,451 - train -  39/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.01/traces_brightclust_Nov2020_D0.6_set010.csv
    2022-02-28 14:19:41,863 - train -  40/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set007.csv
    2022-02-28 14:19:47,487 - train -  41/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set006.csv
    2022-02-28 14:19:53,197 - train -  42/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set002.csv
    2022-02-28 14:20:00,405 - train -  43/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set006.csv
    2022-02-28 14:20:04,607 - train -  44/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set006.csv
    2022-02-28 14:20:08,353 - train -  45/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set010.csv
    2022-02-28 14:20:12,307 - train -  46/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.01/traces_brightclust_Nov2020_D0.08_set007.csv
    2022-02-28 14:20:16,444 - train -  47/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set002.csv
    2022-02-28 14:20:20,213 - train -  48/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set009.csv
    2022-02-28 14:20:23,834 - train -  1/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/0.1/traces_brightclust_Nov2020_D1.0_set009.csv
    2022-02-28 14:20:30,953 - train -  2/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/50/1.0/traces_brightclust_Nov2020_D50_set007.csv
    2022-02-28 14:20:36,797 - train -  3/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.4/0.1/traces_brightclust_Nov2020_D0.4_set009.csv
    2022-02-28 14:20:48,524 - train -  4/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/1.0/traces_brightclust_Nov2020_D3.0_set009.csv
    2022-02-28 14:20:57,745 - train -  5/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.1/0.01/traces_brightclust_Nov2020_D0.1_set008.csv
    2022-02-28 14:21:01,411 - train -  6/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/10/0.01/traces_brightclust_Nov2020_D10_set008.csv
    2022-02-28 14:21:13,337 - train -  7/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.08/0.1/traces_brightclust_Nov2020_D0.08_set008.csv
    2022-02-28 14:21:17,317 - train -  8/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
    2022-02-28 14:21:21,005 - train -  9/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/0.01/traces_brightclust_Nov2020_D3.0_set008.csv
    2022-02-28 14:21:24,834 - train -  10/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.2/0.1/traces_brightclust_Nov2020_D0.2_set006.csv
    2022-02-28 14:21:28,718 - train -  11/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.069/0.1/traces_brightclust_Nov2020_D0.069_set006.csv
    2022-02-28 14:21:32,087 - train -  12/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.6/0.1/traces_brightclust_Nov2020_D0.6_set006.csv
    2022-02-28 14:21:32,293 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 4800), (16384, 4800), (16384, 4800)]
    2022-02-28 14:21:32,383 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 1200), (16384, 1200), (16384, 1200)]
    2022-02-28 14:21:32.540303: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operatio
    ns:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    2022-02-28 14:21:32.542865: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-02-28 14:21:32.544861: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-02-28 14:21:32.544960: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-02-28 14:21:32.969057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
    2022-02-28 14:21:32.969130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0
    2022-02-28 14:21:32.969144: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N
    2022-02-28 14:21:32.972037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15403 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, p
    ci bus id: 0000:82:00.0, compute capability: 6.0)
    2022-02-28 14:21:35,138 - train -  number of examples: 4800
    2022-02-28 14:21:35,561 - train -  number of examples: 1200
    2022-02-28 14:21:37,968 - train -  unet: input shape: (None, None, 1), output shape: (None, None, 1)
    2022/02/28 14:21:38 WARNING mlflow.utils.autologging_utils: Encountered unexpected error during tensorflow autologging: Changing param values is not allowed. Param with key='batch_size' was already logged with value='15' for run ID='0cd
    2023eeaf745aca0d3e8ad5e1fc653'. Attempted logging new value 'None'.
    Epoch 1/100
    2022-02-28 14:21:47.612006: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
    2022-02-28 14:21:47.786795: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199895000 Hz
    2022-02-28 14:22:00.556303: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:177] Filling up shuffle buffer (this may take a while): 3118 of 4800
    2022-02-28 14:22:05.887971: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:230] Shuffle buffer filled.
    2022-02-28 14:22:05.992991: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-02-28 14:22:06.329980: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101
    2022-02-28 14:22:06.837590: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-02-28 14:22:07.115604: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    320/320 [==============================] - 80s 158ms/step - loss: 0.6557 - tp0.1: 8858269.0000 - fp0.1: 11367531.0000 - tn0.1: 57039588.0000 - fn0.1: 1377794.0000 - precision0.1: 0.4380 - recall0.1: 0.8654 - tp0.3: 8020446.0000 - fp0.3:
     6844078.0000 - tn0.3: 61563052.0000 - fn0.3: 2215617.0000 - precision0.3: 0.5396 - recall0.3: 0.7835 - tp0.5: 6484149.0000 - fp0.5: 3021931.0000 - tn0.5: 65385196.0000 - fn0.5: 3751914.0000 - precision0.5: 0.6821 - recall0.5: 0.6335 -
    tp0.7: 4743137.0000 - fp0.7: 1127848.0000 - tn0.7: 67279296.0000 - fn0.7: 5492926.0000 - precision0.7: 0.8079 - recall0.7: 0.4634 - tp0.9: 2178331.0000 - fp0.9: 182418.0000 - tn0.9: 68224688.0000 - fn0.9: 8057732.0000 - precision0.9: 0.
    9227 - recall0.9: 0.2128 - accuracy: 0.9139 - auc: 0.9007 - f1: 0.6569 - val_loss: 154.4101 - val_tp0.1: 2567299.0000 - val_fp0.1: 15932078.0000 - val_tn0.1: 1006227.0000 - val_fn0.1: 155196.0000 - val_precision0.1: 0.1388 - val_recall0
    .1: 0.9430 - val_tp0.3: 2564050.0000 - val_fp0.3: 15892674.0000 - val_tn0.3: 1045631.0000 - val_fn0.3: 158445.0000 - val_precision0.3: 0.1389 - val_recall0.3: 0.9418 - val_tp0.5: 2561216.0000 - val_fp0.5: 15863898.0000 - val_tn0.5: 1074
    407.0000 - val_fn0.5: 161279.0000 - val_precision0.5: 0.1390 - val_recall0.5: 0.9408 - val_tp0.7: 2556675.0000 - val_fp0.7: 15829841.0000 - val_tn0.7: 1108464.0000 - val_fn0.7: 165820.0000 - val_precision0.7: 0.1391 - val_recall0.7: 0.9
    391 - val_tp0.9: 2546087.0000 - val_fp0.9: 15767476.0000 - val_tn0.9: 1170829.0000 - val_fn0.9: 176408.0000 - val_precision0.9: 0.1390 - val_recall0.9: 0.9352 - val_accuracy: 0.1849 - val_auc: 0.4998 - val_f1: 0.2422
    
    ...
    
    Epoch 100/100
    320/320 [==============================] - 46s 145ms/step - loss: 0.1275 - tp0.1: 10018370.0000 - fp0.1: 2700999.0000 - tn0.1: 65706144.0000 - fn0.1: 217693.0000 - precision0.1: 0.7876 - recall0.1: 0.9787 - tp0.3: 9775954.0000 - fp0.3:
    1429847.0000 - tn0.3: 66977260.0000 - fn0.3: 460109.0000 - precision0.3: 0.8724 - recall0.3: 0.9551 - tp0.5: 9497895.0000 - fp0.5: 850165.0000 - tn0.5: 67556944.0000 - fn0.5: 738168.0000 - precision0.5: 0.9178 - recall0.5: 0.9279 - tp0.
    7: 9083449.0000 - fp0.7: 444475.0000 - tn0.7: 67962624.0000 - fn0.7: 1152614.0000 - precision0.7: 0.9534 - recall0.7: 0.8874 - tp0.9: 8160305.0000 - fp0.9: 123032.0000 - tn0.9: 68284080.0000 - fn0.9: 2075758.0000 - precision0.9: 0.9851
    - recall0.9: 0.7972 - accuracy: 0.9798 - auc: 0.9876 - f1: 0.9228 - val_loss: 0.1678 - val_tp0.1: 2639044.0000 - val_fp0.1: 807650.0000 - val_tn0.1: 16130655.0000 - val_fn0.1: 83451.0000 - val_precision0.1: 0.7657 - val_recall0.1: 0.969
    3 - val_tp0.3: 2571140.0000 - val_fp0.3: 452176.0000 - val_tn0.3: 16486129.0000 - val_fn0.3: 151355.0000 - val_precision0.3: 0.8504 - val_recall0.3: 0.9444 - val_tp0.5: 2500729.0000 - val_fp0.5: 291947.0000 - val_tn0.5: 16646358.0000 -
    val_fn0.5: 221766.0000 - val_precision0.5: 0.8955 - val_recall0.5: 0.9185 - val_tp0.7: 2397916.0000 - val_fp0.7: 172157.0000 - val_tn0.7: 16766148.0000 - val_fn0.7: 324579.0000 - val_precision0.7: 0.9330 - val_recall0.7: 0.8808 - val_tp
    0.9: 2174652.0000 - val_fp0.9: 58658.0000 - val_tn0.9: 16879648.0000 - val_fn0.9: 547843.0000 - val_precision0.9: 0.9737 - val_recall0.9: 0.7988 - val_accuracy: 0.9739 - val_auc: 0.9818 - val_f1: 0.9069
    2022-02-28 15:49:04.737117: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
    2022/02/28 15:49:22 INFO mlflow.projects: === Run (ID '0cd2023eeaf745aca0d3e8ad5e1fc653') succeeded ===
    
    • name of run: 0cd2023eeaf745aca0d3e8ad5e1fc653
    • metrics after 100th epoch:
      • loss: 0.1275 - lossval: 0.1678
      • precisionval,0.5: 0.8955 - recallval,0.5: 0.9185 - f1val,0.5: 0.9069
      • aucval: 0.9818
  3. run training of (a5b8551144ff46e697a39cd1551e1475, 98cf8cdef9c54b5286e277e75e2ab8c1) with hparams from above
    mlflow run . -e main -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ \
           -P batch_size=20 \
           -P first_filters=78 \
           -P input_size=14000 \
           -P lr_start=0.0584071108418767 \
           -P lr_power=4 \
           -P epochs=100 \
           -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets \
           -P csv_path_val=/beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN \
           -P scaler=standard \
           -P n_levels=4 \
           -P pool_size=4
    
    2022/02/28 17:34:45 INFO mlflow.projects.utils: === Created directory /tmp/tmpwue2ujb5 for downloading remote URIs passed to arguments of type 'path' ===
    2022/02/28 17:34:45 INFO mlflow.projects.backend.local: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe 1>&2 && python src/fluotracify/train
    ing/train.py --fluotracify_path /beegfs/ye53nis/drmed-git/src --batch_size 20 --input_size 14000 --lr_start 0.0584071108418767 --lr_power 4 --epochs 100 --csv_path_train /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets --csv_p
    ath_val /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN --col_per_example 3 --scaler standard --n_levels 4 --first_filters 78 --pool_size 4' in run with ID 'fe81d71c52404ed790b3a32051258da9' ===
    2022-02-28 17:34:58.034002: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-02-28 17:35:10,330 - train -  Python version: 3.9.6 (default, Jul 30 2021, 16:35:19)
    [GCC 7.5.0]
    2022-02-28 17:35:10,331 - train -  Tensorflow version: 2.5.0
    2022-02-28 17:35:10,331 - train -  tf.keras version: 2.5.0
    2022-02-28 17:35:10,331 - train -  Cudnn version: 8
    2022-02-28 17:35:10,331 - train -  Cuda version: 11.2
    2022-02-28 17:35:10.333548: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
    2022-02-28 17:35:10.391240: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-02-28 17:35:10.391388: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-02-28 17:35:10.401553: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-02-28 17:35:10.401669: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    2022-02-28 17:35:10.405228: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
    2022-02-28 17:35:10.407042: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
    2022-02-28 17:35:10.415485: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
    2022-02-28 17:35:10.417900: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
    2022-02-28 17:35:10.419606: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-02-28 17:35:10.422879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-02-28 17:35:10,423 - train -  GPUs: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]. Trying to set memory growth to "True"...
    2022-02-28 17:35:10,423 - train -  Setting memory growth successful.
    2022-02-28 17:35:16,324 - train -  1/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set004.csv
    2022-02-28 17:35:30,676 - train -  2/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set002.csv
    2022-02-28 17:35:34,283 - train -  3/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set001.csv
    2022-02-28 17:35:38,780 - train -  4/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set007.csv
    2022-02-28 17:35:43,054 - train -  5/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
    2022-02-28 17:35:46,234 - train -  6/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/1.0/traces_brightclust_Nov2020_D10_set010.csv
    2022-02-28 17:35:50,337 - train -  7/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set004.csv
    2022-02-28 17:35:53,421 - train -  8/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.1/traces_brightclust_Nov2020_D3.0_set010.csv
    2022-02-28 17:35:59,004 - train -  9/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set009.csv
    2022-02-28 17:36:02,669 - train -  10/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set003.csv
    2022-02-28 17:36:06,267 - train -  11/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set007.csv
    2022-02-28 17:36:14,414 - train -  12/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
    2022-02-28 17:36:18,640 - train -  13/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set003.csv
    2022-02-28 17:36:21,742 - train -  14/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set002.csv
    2022-02-28 17:36:25,004 - train -  15/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set005.csv
    2022-02-28 17:36:28,518 - train -  16/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/1.0/traces_brightclust_Nov2020_D0.069_set009.csv
    2022-02-28 17:36:31,957 - train -  17/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set004.csv
    2022-02-28 17:36:35,265 - train -  18/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set010.csv
    2022-02-28 17:36:38,670 - train -  19/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set009.csv
    2022-02-28 17:36:42,309 - train -  20/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.01/traces_brightclust_Nov2020_D1.0_set010.csv
    2022-02-28 17:36:45,973 - train -  21/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set007.csv
    2022-02-28 17:36:53,886 - train -  22/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set010.csv
    2022-02-28 17:36:57,387 - train -  23/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set001.csv
    2022-02-28 17:37:00,889 - train -  24/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set003.csv
    2022-02-28 17:37:03,901 - train -  25/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set003.csv
    2022-02-28 17:37:08,741 - train -  26/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set004.csv
    2022-02-28 17:37:11,797 - train -  27/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set001.csv
    2022-02-28 17:37:15,549 - train -  28/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set005.csv
    2022-02-28 17:37:19,909 - train -  29/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set006.csv
    2022-02-28 17:37:32,956 - train -  30/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set004.csv
    2022-02-28 17:37:47,613 - train -  31/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set006.csv
    2022-02-28 17:37:51,591 - train -  32/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set004.csv
    2022-02-28 17:37:59,140 - train -  33/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set005.csv
    2022-02-28 17:38:03,933 - train -  34/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set003.csv
    2022-02-28 17:38:07,415 - train -  35/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.01/traces_brightclust_Nov2020_D0.2_set003.csv
    2022-02-28 17:38:15,509 - train -  36/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.01/traces_brightclust_Nov2020_D50_set006.csv
    2022-02-28 17:38:20,075 - train -  37/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set004.csv
    2022-02-28 17:38:24,146 - train -  38/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set004.csv
    2022-02-28 17:38:38,735 - train -  39/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.01/traces_brightclust_Nov2020_D0.6_set010.csv
    2022-02-28 17:38:41,894 - train -  40/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set007.csv
    2022-02-28 17:38:45,710 - train -  41/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set006.csv
    2022-02-28 17:38:49,786 - train -  42/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set002.csv
    2022-02-28 17:38:53,505 - train -  43/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set006.csv
    2022-02-28 17:38:58,205 - train -  44/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set006.csv
    2022-02-28 17:39:02,031 - train -  45/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set010.csv
    2022-02-28 17:39:06,190 - train -  46/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.01/traces_brightclust_Nov2020_D0.08_set007.csv
    2022-02-28 17:39:09,521 - train -  47/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set002.csv
    2022-02-28 17:39:14,557 - train -  48/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set009.csv
    2022-02-28 17:39:18,060 - train -  1/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/0.1/traces_brightclust_Nov2020_D1.0_set009.csv
    2022-02-28 17:39:31,842 - train -  2/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/50/1.0/traces_brightclust_Nov2020_D50_set007.csv
    2022-02-28 17:39:51,088 - train -  3/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.4/0.1/traces_brightclust_Nov2020_D0.4_set009.csv
    2022-02-28 17:40:02,789 - train -  4/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/1.0/traces_brightclust_Nov2020_D3.0_set009.csv
    2022-02-28 17:40:06,696 - train -  5/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.1/0.01/traces_brightclust_Nov2020_D0.1_set008.csv
    2022-02-28 17:40:10,118 - train -  6/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/10/0.01/traces_brightclust_Nov2020_D10_set008.csv
    2022-02-28 17:40:24,593 - train -  7/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.08/0.1/traces_brightclust_Nov2020_D0.08_set008.csv
    2022-02-28 17:40:28,308 - train -  8/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
    2022-02-28 17:40:37,143 - train -  9/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/0.01/traces_brightclust_Nov2020_D3.0_set008.csv
    2022-02-28 17:40:41,060 - train -  10/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.2/0.1/traces_brightclust_Nov2020_D0.2_set006.csv
    2022-02-28 17:40:59,911 - train -  11/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.069/0.1/traces_brightclust_Nov2020_D0.069_set006.csv
    2022-02-28 17:41:19,363 - train -  12/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.6/0.1/traces_brightclust_Nov2020_D0.6_set006.csv
    2022-02-28 17:41:19,570 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 4800), (16384, 4800), (16384, 4800)]
    2022-02-28 17:41:19,659 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 1200), (16384, 1200), (16384, 1200)]
    2022-02-28 17:41:19.824149: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operatio
    ns:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    2022-02-28 17:41:19.827704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-02-28 17:41:19.831605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-02-28 17:41:19.831748: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-02-28 17:41:20.290008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
    2022-02-28 17:41:20.290081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0
    2022-02-28 17:41:20.290096: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N
    2022-02-28 17:41:20.292984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15403 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, p
    ci bus id: 0000:82:00.0, compute capability: 6.0)
    2022-02-28 17:41:22,312 - train -  number of examples: 4800
    2022-02-28 17:41:22,698 - train -  number of examples: 1200
    2022-02-28 17:41:26,203 - train -  unet: input shape: (None, None, 1), output shape: (None, None, 1)
    2022/02/28 17:41:26 WARNING mlflow.utils.autologging_utils: Encountered unexpected error during tensorflow autologging: Changing param values is not allowed. Param with key='batch_size' was already logged with value='20' for run ID='fe8
    1d71c52404ed790b3a32051258da9'. Attempted logging new value 'None'.
    Epoch 1/100
    2022-02-28 17:41:34.119597: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
    2022-02-28 17:41:34.266024: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199895000 Hz
    2022-02-28 17:41:42.993902: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-02-28 17:41:43.305956: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101
    2022-02-28 17:41:43.787015: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-02-28 17:41:44.068827: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    240/240 [==============================] - 146s 532ms/step - loss: 0.8392 - tp0.1: 8747285.0000 - fp0.1: 10445142.0000 - tn0.1: 57962008.0000 - fn0.1: 1488778.0000 - precision0.1: 0.4558 - recall0.1: 0.8546 - tp0.3: 7896000.0000 - fp0.3
    : 5233157.0000 - tn0.3: 63173976.0000 - fn0.3: 2340063.0000 - precision0.3: 0.6014 - recall0.3: 0.7714 - tp0.5: 6575708.0000 - fp0.5: 2263621.0000 - tn0.5: 66143520.0000 - fn0.5: 3660355.0000 - precision0.5: 0.7439 - recall0.5: 0.6424 -
     tp0.7: 5154156.0000 - fp0.7: 920133.0000 - tn0.7: 67487000.0000 - fn0.7: 5081907.0000 - precision0.7: 0.8485 - recall0.7: 0.5035 - tp0.9: 3283109.0000 - fp0.9: 213358.0000 - tn0.9: 68193792.0000 - fn0.9: 6952954.0000 - precision0.9: 0.
    9390 - recall0.9: 0.3207 - accuracy: 0.9247 - auc: 0.9073 - f1: 0.6894 - val_loss: 1.7768 - val_tp0.1: 2543119.0000 - val_fp0.1: 9378664.0000 - val_tn0.1: 7559641.0000 - val_fn0.1: 179376.0000 - val_precision0.1: 0.2133 - val_recall0.1:
     0.9341 - val_tp0.3: 2504900.0000 - val_fp0.3: 7161614.0000 - val_tn0.3: 9776691.0000 - val_fn0.3: 217595.0000 - val_precision0.3: 0.2591 - val_recall0.3: 0.9201 - val_tp0.5: 2470794.0000 - val_fp0.5: 5614724.0000 - val_tn0.5: 11323581.
    0000 - val_fn0.5: 251701.0000 - val_precision0.5: 0.3056 - val_recall0.5: 0.9075 - val_tp0.7: 2416278.0000 - val_fp0.7: 4187348.0000 - val_tn0.7: 12750957.0000 - val_fn0.7: 306217.0000 - val_precision0.7: 0.3659 - val_recall0.7: 0.8875
    - val_tp0.9: 2331726.0000 - val_fp0.9: 2968486.0000 - val_tn0.9: 13969819.0000 - val_fn0.9: 390769.0000 - val_precision0.9: 0.4399 - val_recall0.9: 0.8565 - val_accuracy: 0.7016 - val_auc: 0.8807 - val_f1: 0.4572
    
    ...
    
    Epoch 100/100
    240/240 [==============================] - 124s 515ms/step - loss: 0.0941 - tp0.1: 10073753.0000 - fp0.1: 1908290.0000 - tn0.1: 66498824.0000 - fn0.1: 162310.0000 - precision0.1: 0.8407 - recall0.1: 0.9841 - tp0.3: 9929412.0000 - fp0.3:
     1163791.0000 - tn0.3: 67243336.0000 - fn0.3: 306651.0000 - precision0.3: 0.8951 - recall0.3: 0.9700 - tp0.5: 9693885.0000 - fp0.5: 672284.0000 - tn0.5: 67734840.0000 - fn0.5: 542178.0000 - precision0.5: 0.9351 - recall0.5: 0.9470 - tp0
    .7: 9345709.0000 - fp0.7: 339542.0000 - tn0.7: 68067608.0000 - fn0.7: 890354.0000 - precision0.7: 0.9649 - recall0.7: 0.9130 - tp0.9: 8608433.0000 - fp0.9: 86583.0000 - tn0.9: 68320560.0000 - fn0.9: 1627630.0000 - precision0.9: 0.9900 -
     recall0.9: 0.8410 - accuracy: 0.9846 - auc: 0.9912 - f1: 0.9411 - val_loss: 0.1372 - val_tp0.1: 2648989.0000 - val_fp0.1: 600014.0000 - val_tn0.1: 16338291.0000 - val_fn0.1: 73506.0000 - val_precision0.1: 0.8153 - val_recall0.1: 0.9730
     - val_tp0.3: 2604739.0000 - val_fp0.3: 372047.0000 - val_tn0.3: 16566258.0000 - val_fn0.3: 117756.0000 - val_precision0.3: 0.8750 - val_recall0.3: 0.9567 - val_tp0.5: 2542400.0000 - val_fp0.5: 225996.0000 - val_tn0.5: 16712309.0000 - v
    al_fn0.5: 180095.0000 - val_precision0.5: 0.9184 - val_recall0.5: 0.9338 - val_tp0.7: 2459692.0000 - val_fp0.7: 126768.0000 - val_tn0.7: 16811536.0000 - val_fn0.7: 262803.0000 - val_precision0.7: 0.9510 - val_recall0.7: 0.9035 - val_tp0
    .9: 2288921.0000 - val_fp0.9: 43449.0000 - val_tn0.9: 16894856.0000 - val_fn0.9: 433574.0000 - val_precision0.9: 0.9814 - val_recall0.9: 0.8407 - val_accuracy: 0.9793 - val_auc: 0.9849 - val_f1: 0.9260
    2022-02-28 21:13:24.522272: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
    2022/02/28 21:13:37 INFO mlflow.projects: === Run (ID 'fe81d71c52404ed790b3a32051258da9') succeeded ===
    (tf) [ye53nis@node128 drmed-git]$
    
    • name of run: fe81d71c52404ed790b3a32051258da9
    • metrics after 100th epoch:
      • precisionval,0.5: 0.9184 - recallval,0.5: 0.9338 - f1val,0.5: 0.9260
      • aucval: 0.9849
  4. run training of (00f2635d9fa2463c9a066722163405be, d0a8e1748b194f3290d471b6b44f19f8) with hparams from above
    mlflow run . -e main -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ \
           -P batch_size=28 \
           -P first_filters=6 \
           -P input_size=14000 \
           -P lr_start=0.0553313915596308 \
           -P lr_power=1 \
           -P epochs=100 \
           -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets \
           -P csv_path_val=/beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN \
           -P scaler=minmax \
           -P n_levels=5 \
           -P pool_size=4
    
    2022/03/01 01:02:32 INFO mlflow.projects.utils: === Created directory /tmp/tmpscsw8dai for downloading remote URIs passed to arguments of type 'path' ===
    2022/03/01 01:02:32 INFO mlflow.projects.backend.local: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe 1>&2 && python src/fluotracify/train
    ing/train.py --fluotracify_path /beegfs/ye53nis/drmed-git/src --batch_size 28 --input_size 14000 --lr_start 0.0553313915596308 --lr_power 1 --epochs 100 --csv_path_train /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets --csv_p
    ath_val /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN --col_per_example 3 --scaler minmax --n_levels 5 --first_filters 6 --pool_size 4' in run with ID 'ff67be0b68e540a9a29a36a2d0c7a5be' ===
    2022-03-01 01:02:49.062309: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-01 01:03:02,012 - train -  Python version: 3.9.6 (default, Jul 30 2021, 16:35:19)
    [GCC 7.5.0]
    2022-03-01 01:03:02,013 - train -  Tensorflow version: 2.5.0
    2022-03-01 01:03:02,013 - train -  tf.keras version: 2.5.0
    2022-03-01 01:03:02,013 - train -  Cudnn version: 8
    2022-03-01 01:03:02,013 - train -  Cuda version: 11.2
    2022-03-01 01:03:02.017849: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
    2022-03-01 01:03:02.070568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-03-01 01:03:02.070675: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-01 01:03:02.081043: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-03-01 01:03:02.081139: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    2022-03-01 01:03:02.085346: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
    2022-03-01 01:03:02.088350: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
    2022-03-01 01:03:02.097715: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
    2022-03-01 01:03:02.100867: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
    2022-03-01 01:03:02.103117: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-03-01 01:03:02.106443: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-03-01 01:03:02,106 - train -  GPUs: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]. Trying to set memory growth to "True"...
    2022-03-01 01:03:02,107 - train -  Setting memory growth successful.
    2022-03-01 01:03:08,710 - train -  1/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set004.csv
    2022-03-01 01:03:17,500 - train -  2/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set002.csv
    2022-03-01 01:03:21,523 - train -  3/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set001.csv
    2022-03-01 01:03:25,810 - train -  4/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set007.csv
    2022-03-01 01:03:29,726 - train -  5/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
    2022-03-01 01:03:35,702 - train -  6/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/1.0/traces_brightclust_Nov2020_D10_set010.csv
    2022-03-01 01:03:39,927 - train -  7/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set004.csv
    2022-03-01 01:03:44,978 - train -  8/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.1/traces_brightclust_Nov2020_D3.0_set010.csv
    2022-03-01 01:03:49,851 - train -  9/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set009.csv
    2022-03-01 01:03:54,065 - train -  10/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set003.csv
    2022-03-01 01:03:57,914 - train -  11/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set007.csv
    2022-03-01 01:04:09,148 - train -  12/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
    2022-03-01 01:04:14,246 - train -  13/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set003.csv
    2022-03-01 01:04:17,776 - train -  14/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set002.csv
    2022-03-01 01:04:24,334 - train -  15/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set005.csv
    2022-03-01 01:04:28,494 - train -  16/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/1.0/traces_brightclust_Nov2020_D0.069_set009.csv
    2022-03-01 01:04:31,978 - train -  17/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set004.csv
    2022-03-01 01:04:35,535 - train -  18/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set010.csv
    2022-03-01 01:04:39,104 - train -  19/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set009.csv
    2022-03-01 01:04:42,753 - train -  20/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.01/traces_brightclust_Nov2020_D1.0_set010.csv
    2022-03-01 01:04:46,786 - train -  21/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set007.csv
    2022-03-01 01:04:50,549 - train -  22/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set010.csv
    2022-03-01 01:05:10,273 - train -  23/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set001.csv
    2022-03-01 01:05:14,332 - train -  24/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set003.csv
    2022-03-01 01:05:17,829 - train -  25/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set003.csv
    2022-03-01 01:05:29,382 - train -  26/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set004.csv
    2022-03-01 01:05:32,892 - train -  27/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set001.csv
    2022-03-01 01:05:37,614 - train -  28/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set005.csv
    2022-03-01 01:05:41,478 - train -  29/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set006.csv
    2022-03-01 01:05:55,816 - train -  30/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set004.csv
    2022-03-01 01:06:06,797 - train -  31/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set006.csv
    2022-03-01 01:06:10,595 - train -  32/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set004.csv
    2022-03-01 01:06:15,368 - train -  33/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set005.csv
    2022-03-01 01:06:20,784 - train -  34/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set003.csv
    2022-03-01 01:06:26,970 - train -  35/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.01/traces_brightclust_Nov2020_D0.2_set003.csv
    2022-03-01 01:06:31,521 - train -  36/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.01/traces_brightclust_Nov2020_D50_set006.csv
    2022-03-01 01:06:35,363 - train -  37/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set004.csv
    2022-03-01 01:06:39,107 - train -  38/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set004.csv
    2022-03-01 01:07:01,475 - train -  39/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.01/traces_brightclust_Nov2020_D0.6_set010.csv
    2022-03-01 01:07:05,191 - train -  40/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set007.csv
    2022-03-01 01:07:08,941 - train -  41/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set006.csv
    2022-03-01 01:07:13,043 - train -  42/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set002.csv
    2022-03-01 01:07:16,874 - train -  43/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set006.csv
    2022-03-01 01:07:20,478 - train -  44/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set006.csv
    2022-03-01 01:07:24,571 - train -  45/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set010.csv
    2022-03-01 01:07:28,090 - train -  46/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.01/traces_brightclust_Nov2020_D0.08_set007.csv
    2022-03-01 01:07:31,811 - train -  47/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set002.csv
    2022-03-01 01:07:35,574 - train -  48/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set009.csv
    2022-03-01 01:07:39,526 - train -  1/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/0.1/traces_brightclust_Nov2020_D1.0_set009.csv
    2022-03-01 01:07:49,139 - train -  2/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/50/1.0/traces_brightclust_Nov2020_D50_set007.csv
    2022-03-01 01:08:06,079 - train -  3/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.4/0.1/traces_brightclust_Nov2020_D0.4_set009.csv
    2022-03-01 01:08:17,384 - train -  4/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/1.0/traces_brightclust_Nov2020_D3.0_set009.csv
    2022-03-01 01:08:21,021 - train -  5/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.1/0.01/traces_brightclust_Nov2020_D0.1_set008.csv
    2022-03-01 01:08:24,646 - train -  6/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/10/0.01/traces_brightclust_Nov2020_D10_set008.csv
    2022-03-01 01:08:33,131 - train -  7/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.08/0.1/traces_brightclust_Nov2020_D0.08_set008.csv
    2022-03-01 01:08:37,919 - train -  8/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
    2022-03-01 01:08:41,923 - train -  9/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/0.01/traces_brightclust_Nov2020_D3.0_set008.csv
    2022-03-01 01:08:45,386 - train -  10/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.2/0.1/traces_brightclust_Nov2020_D0.2_set006.csv
    2022-03-01 01:08:50,064 - train -  11/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.069/0.1/traces_brightclust_Nov2020_D0.069_set006.csv
    2022-03-01 01:08:54,537 - train -  12/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.6/0.1/traces_brightclust_Nov2020_D0.6_set006.csv
    2022-03-01 01:08:54,754 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 4800), (16384, 4800), (16384, 4800)]
    2022-03-01 01:08:54,844 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 1200), (16384, 1200), (16384, 1200)]
    2022-03-01 01:08:54.997694: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operatio
    ns:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    2022-03-01 01:08:55.000062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-03-01 01:08:55.002040: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-03-01 01:08:55.002135: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-01 01:08:55.428415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
    2022-03-01 01:08:55.428487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0
    2022-03-01 01:08:55.428501: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N
    2022-03-01 01:08:55.431349: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15403 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, p
    ci bus id: 0000:82:00.0, compute capability: 6.0)
    2022-03-01 01:08:57,489 - train -  number of examples: 4800
    2022-03-01 01:08:57,859 - train -  number of examples: 1200
    2022-03-01 01:09:00,816 - train -  unet: input shape: (None, None, 1), output shape: (None, None, 1)
    2022/03/01 01:09:00 WARNING mlflow.utils.autologging_utils: Encountered unexpected error during tensorflow autologging: Changing param values is not allowed. Param with key='batch_size' was already logged with value='28' for run ID='ff6
    7be0b68e540a9a29a36a2d0c7a5be'. Attempted logging new value 'None'.
    Epoch 1/100
    2022-03-01 01:09:09.470355: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
    2022-03-01 01:09:09.627221: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199895000 Hz
    2022-03-01 01:09:19.253073: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-03-01 01:09:19.568134: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101
    2022-03-01 01:09:20.032500: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-03-01 01:09:20.318967: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    171/171 [==============================] - 38s 111ms/step - loss: 0.6328 - tp0.1: 9097064.0000 - fp0.1: 12967912.0000 - tn0.1: 55269208.0000 - fn0.1: 1112405.0000 - precision0.1: 0.4123 - recall0.1: 0.8910 - tp0.3: 8278749.0000 - fp0.3:
     7308225.0000 - tn0.3: 60928896.0000 - fn0.3: 1930720.0000 - precision0.3: 0.5311 - recall0.3: 0.8109 - tp0.5: 6546144.0000 - fp0.5: 2745102.0000 - tn0.5: 65492008.0000 - fn0.5: 3663325.0000 - precision0.5: 0.7045 - recall0.5: 0.6412 -
    tp0.7: 5069917.0000 - fp0.7: 1036309.0000 - tn0.7: 67200816.0000 - fn0.7: 5139552.0000 - precision0.7: 0.8303 - recall0.7: 0.4966 - tp0.9: 2806365.0000 - fp0.9: 200542.0000 - tn0.9: 68036592.0000 - fn0.9: 7403104.0000 - precision0.9: 0.
    9333 - recall0.9: 0.2749 - accuracy: 0.9183 - auc: 0.9138 - f1: 0.6714 - val_loss: 100.5511 - val_tp0.1: 2677947.0000 - val_fp0.1: 16589637.0000 - val_tn0.1: 0.0000e+00 - val_fn0.1: 0.0000e+00 - val_precision0.1: 0.1390 - val_recall0.1:
     1.0000 - val_tp0.3: 2677947.0000 - val_fp0.3: 16589563.0000 - val_tn0.3: 74.0000 - val_fn0.3: 0.0000e+00 - val_precision0.3: 0.1390 - val_recall0.3: 1.0000 - val_tp0.5: 2677947.0000 - val_fp0.5: 16589315.0000 - val_tn0.5: 322.0000 - va
    l_fn0.5: 0.0000e+00 - val_precision0.5: 0.1390 - val_recall0.5: 1.0000 - val_tp0.7: 2677947.0000 - val_fp0.7: 16589042.0000 - val_tn0.7: 595.0000 - val_fn0.7: 0.0000e+00 - val_precision0.7: 0.1390 - val_recall0.7: 1.0000 - val_tp0.9: 26
    77947.0000 - val_fp0.9: 16588561.0000 - val_tn0.9: 1076.0000 - val_fn0.9: 0.0000e+00 - val_precision0.9: 0.1390 - val_recall0.9: 1.0000 - val_accuracy: 0.1390 - val_auc: 0.5001 - val_f1: 0.2441
    
    ...
    
    Epoch 100/100
    171/171 [==============================] - 15s 89ms/step - loss: 0.0890 - tp0.1: 10079569.0000 - fp0.1: 1892418.0000 - tn0.1: 66332504.0000 - fn0.1: 142100.0000 - precision0.1: 0.8419 - recall0.1: 0.9861 - tp0.3: 9939857.0000 - fp0.3: 1
    140232.0000 - tn0.3: 67084692.0000 - fn0.3: 281812.0000 - precision0.3: 0.8971 - recall0.3: 0.9724 - tp0.5: 9703286.0000 - fp0.5: 647685.0000 - tn0.5: 67577248.0000 - fn0.5: 518383.0000 - precision0.5: 0.9374 - recall0.5: 0.9493 - tp0.7
    : 9385401.0000 - fp0.7: 340758.0000 - tn0.7: 67884160.0000 - fn0.7: 836268.0000 - precision0.7: 0.9650 - recall0.7: 0.9182 - tp0.9: 8688940.0000 - fp0.9: 89880.0000 - tn0.9: 68135024.0000 - fn0.9: 1532729.0000 - precision0.9: 0.9898 - r
    ecall0.9: 0.8501 - accuracy: 0.9851 - auc: 0.9922 - f1: 0.9433 - val_loss: 0.1286 - val_tp0.1: 2593533.0000 - val_fp0.1: 546329.0000 - val_tn0.1: 16060347.0000 - val_fn0.1: 67375.0000 - val_precision0.1: 0.8260 - val_recall0.1: 0.9747 -
     val_tp0.3: 2548361.0000 - val_fp0.3: 333669.0000 - val_tn0.3: 16273007.0000 - val_fn0.3: 112547.0000 - val_precision0.3: 0.8842 - val_recall0.3: 0.9577 - val_tp0.5: 2492439.0000 - val_fp0.5: 208035.0000 - val_tn0.5: 16398641.0000 - val
    _fn0.5: 168469.0000 - val_precision0.5: 0.9230 - val_recall0.5: 0.9367 - val_tp0.7: 2414912.0000 - val_fp0.7: 119511.0000 - val_tn0.7: 16487165.0000 - val_fn0.7: 245996.0000 - val_precision0.7: 0.9528 - val_recall0.7: 0.9076 - val_tp0.9
    : 2250009.0000 - val_fp0.9: 39723.0000 - val_tn0.9: 16566953.0000 - val_fn0.9: 410899.0000 - val_precision0.9: 0.9827 - val_recall0.9: 0.8456 - val_accuracy: 0.9805 - val_auc: 0.9859 - val_f1: 0.9298
    2022-03-01 01:40:07.449069: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
    2022/03/01 01:40:21 INFO mlflow.projects: === Run (ID 'ff67be0b68e540a9a29a36a2d0c7a5be') succeeded ===
    
    
    • name of run: ff67be0b68e540a9a29a36a2d0c7a5be
    • metrics after 100th epoch:
      • loss: 0.0890 vs lossval: 0.1286
      • precisionval,0.5: 0.9230 - recallval,0.5: 0.9367 - f1val,0.5: 0.9298
      • aucval: 0.9859
  5. run training of (5604d43c1ece461b8e6eaa0dfb65d6dc, 3612536a77f34f22bc83d1d809140aa6) with hparams from above
    mlflow run . -e main -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ \
           -P batch_size=20 \
           -P first_filters=128 \
           -P input_size=14000 \
           -P lr_start=0.043549707353273 \
           -P lr_power=1 \
           -P epochs=100 \
           -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets \
           -P csv_path_val=/beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN \
           -P scaler=standard \
           -P n_levels=3 \
           -P pool_size=4
    
    2022/03/01 12:32:44 INFO mlflow.projects.utils: === Created directory /tmp/tmpuaqmbl4a for downloading remote URIs passed to arguments of type 'path' ===
    2022/03/01 12:32:44 INFO mlflow.projects.backend.local: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe 1>&2 && python src/fluotracify/train
    ing/train.py --fluotracify_path /beegfs/ye53nis/drmed-git/src --batch_size 20 --input_size 14000 --lr_start 0.043549707353273 --lr_power 1 --epochs 100 --csv_path_train /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets --csv_pa
    th_val /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN --col_per_example 3 --scaler standard --n_levels 3 --first_filters 128 --pool_size 4' in run with ID '19e3e786e1bc4e2b93856f5dc9de8216' ===
    2022-03-01 12:32:58.514798: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-01 12:33:09,490 - train -  Python version: 3.9.6 (default, Jul 30 2021, 16:35:19)
    [GCC 7.5.0]
    2022-03-01 12:33:09,490 - train -  Tensorflow version: 2.5.0
    2022-03-01 12:33:09,490 - train -  tf.keras version: 2.5.0
    2022-03-01 12:33:09,490 - train -  Cudnn version: 8
    2022-03-01 12:33:09,490 - train -  Cuda version: 11.2
    2022-03-01 12:33:09.493303: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
    2022-03-01 12:33:09.550241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-03-01 12:33:09.550350: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-01 12:33:09.561190: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-03-01 12:33:09.561299: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    2022-03-01 12:33:09.565126: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
    2022-03-01 12:33:09.567931: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
    2022-03-01 12:33:09.576981: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
    2022-03-01 12:33:09.597335: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
    2022-03-01 12:33:09.599375: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-03-01 12:33:09.602568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-03-01 12:33:09,602 - train -  GPUs: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]. Trying to set memory growth to "True"...
    2022-03-01 12:33:09,603 - train -  Setting memory growth successful.
    2022-03-01 12:33:15,918 - train -  1/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set004.csv
    2022-03-01 12:33:19,510 - train -  2/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set002.csv
    2022-03-01 12:33:23,436 - train -  3/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set001.csv
    2022-03-01 12:33:27,059 - train -  4/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set007.csv
    2022-03-01 12:33:31,773 - train -  5/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
    2022-03-01 12:33:35,876 - train -  6/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/1.0/traces_brightclust_Nov2020_D10_set010.csv
    2022-03-01 12:33:39,462 - train -  7/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set004.csv
    2022-03-01 12:33:42,942 - train -  8/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.1/traces_brightclust_Nov2020_D3.0_set010.csv
    2022-03-01 12:33:46,757 - train -  9/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set009.csv
    2022-03-01 12:33:51,064 - train -  10/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set003.csv
    2022-03-01 12:33:56,528 - train -  11/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set007.csv
    2022-03-01 12:34:00,572 - train -  12/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
    2022-03-01 12:34:04,069 - train -  13/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set003.csv
    2022-03-01 12:34:07,965 - train -  14/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set002.csv
    2022-03-01 12:34:18,633 - train -  15/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set005.csv
    2022-03-01 12:34:21,910 - train -  16/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/1.0/traces_brightclust_Nov2020_D0.069_set009.csv
    2022-03-01 12:34:25,641 - train -  17/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set004.csv
    2022-03-01 12:34:30,386 - train -  18/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set010.csv
    2022-03-01 12:34:34,226 - train -  19/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set009.csv
    2022-03-01 12:34:37,740 - train -  20/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.01/traces_brightclust_Nov2020_D1.0_set010.csv
    2022-03-01 12:34:43,348 - train -  21/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set007.csv
    2022-03-01 12:34:46,797 - train -  22/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set010.csv
    2022-03-01 12:34:50,435 - train -  23/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set001.csv
    2022-03-01 12:34:54,778 - train -  24/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set003.csv
    2022-03-01 12:34:58,764 - train -  25/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set003.csv
    2022-03-01 12:35:02,693 - train -  26/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set004.csv
    2022-03-01 12:35:15,387 - train -  27/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set001.csv
    2022-03-01 12:35:19,459 - train -  28/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set005.csv
    2022-03-01 12:35:23,164 - train -  29/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set006.csv
    2022-03-01 12:35:27,345 - train -  30/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set004.csv
    2022-03-01 12:35:30,844 - train -  31/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set006.csv
    2022-03-01 12:35:34,995 - train -  32/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set004.csv
    2022-03-01 12:35:38,565 - train -  33/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set005.csv
    2022-03-01 12:35:43,358 - train -  34/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set003.csv
    2022-03-01 12:35:48,209 - train -  35/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.01/traces_brightclust_Nov2020_D0.2_set003.csv
    2022-03-01 12:35:52,885 - train -  36/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.01/traces_brightclust_Nov2020_D50_set006.csv
    2022-03-01 12:35:57,900 - train -  37/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set004.csv
    2022-03-01 12:36:01,708 - train -  38/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set004.csv
    2022-03-01 12:36:05,386 - train -  39/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.01/traces_brightclust_Nov2020_D0.6_set010.csv
    2022-03-01 12:36:09,008 - train -  40/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set007.csv
    2022-03-01 12:36:13,039 - train -  41/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set006.csv
    2022-03-01 12:36:16,946 - train -  42/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set002.csv
    2022-03-01 12:36:20,517 - train -  43/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set006.csv
    2022-03-01 12:36:24,115 - train -  44/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set006.csv
    2022-03-01 12:36:27,768 - train -  45/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set010.csv
    2022-03-01 12:36:32,136 - train -  46/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.01/traces_brightclust_Nov2020_D0.08_set007.csv
    2022-03-01 12:36:36,005 - train -  47/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set002.csv
    2022-03-01 12:36:39,799 - train -  48/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set009.csv
    2022-03-01 12:36:43,465 - train -  1/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/0.1/traces_brightclust_Nov2020_D1.0_set009.csv
    2022-03-01 12:36:47,166 - train -  2/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/50/1.0/traces_brightclust_Nov2020_D50_set007.csv
    2022-03-01 12:36:50,897 - train -  3/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.4/0.1/traces_brightclust_Nov2020_D0.4_set009.csv
    2022-03-01 12:36:56,421 - train -  4/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/1.0/traces_brightclust_Nov2020_D3.0_set009.csv
    2022-03-01 12:36:59,926 - train -  5/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.1/0.01/traces_brightclust_Nov2020_D0.1_set008.csv
    2022-03-01 12:37:03,654 - train -  6/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/10/0.01/traces_brightclust_Nov2020_D10_set008.csv
    2022-03-01 12:37:07,675 - train -  7/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.08/0.1/traces_brightclust_Nov2020_D0.08_set008.csv
    2022-03-01 12:37:11,784 - train -  8/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
    2022-03-01 12:37:18,953 - train -  9/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/0.01/traces_brightclust_Nov2020_D3.0_set008.csv
    2022-03-01 12:37:22,467 - train -  10/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.2/0.1/traces_brightclust_Nov2020_D0.2_set006.csv
    2022-03-01 12:37:26,203 - train -  11/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.069/0.1/traces_brightclust_Nov2020_D0.069_set006.csv
    2022-03-01 12:37:31,518 - train -  12/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.6/0.1/traces_brightclust_Nov2020_D0.6_set006.csv
    2022-03-01 12:37:31,732 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 4800), (16384, 4800), (16384, 4800)]
    2022-03-01 12:37:31,822 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 1200), (16384, 1200), (16384, 1200)]
    2022-03-01 12:37:32.029003: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operatio
    ns:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    2022-03-01 12:37:32.031379: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-03-01 12:37:32.033418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-03-01 12:37:32.033500: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-01 12:37:32.469006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
    2022-03-01 12:37:32.469079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0
    2022-03-01 12:37:32.469093: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N
    2022-03-01 12:37:32.471972: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15403 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, p
    ci bus id: 0000:82:00.0, compute capability: 6.0)
    2022-03-01 12:37:34,440 - train -  number of examples: 4800
    2022-03-01 12:37:34,767 - train -  number of examples: 1200
    2022-03-01 12:37:36,587 - train -  unet: input shape: (None, None, 1), output shape: (None, None, 1)
    2022/03/01 12:37:36 WARNING mlflow.utils.autologging_utils: Encountered unexpected error during tensorflow autologging: Changing param values is not allowed. Param with key='batch_size' was already logged with value='20' for run ID='19e
    3e786e1bc4e2b93856f5dc9de8216'. Attempted logging new value 'None'.
    Epoch 1/100
    2022-03-01 12:37:44.731191: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
    2022-03-01 12:37:44.863807: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199895000 Hz
    2022-03-01 12:37:53.275302: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-03-01 12:37:53.580058: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101
    2022-03-01 12:37:54.077952: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-03-01 12:37:54.363640: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    240/240 [==============================] - 220s 838ms/step - loss: 1.1221 - tp0.1: 8540565.0000 - fp0.1: 11192430.0000 - tn0.1: 57214708.0000 - fn0.1: 1695498.0000 - precision0.1: 0.4328 - recall0.1: 0.8344 - tp0.3: 7522117.0000 - fp0.3
    : 4392113.0000 - tn0.3: 64015032.0000 - fn0.3: 2713946.0000 - precision0.3: 0.6314 - recall0.3: 0.7349 - tp0.5: 6672931.0000 - fp0.5: 2324779.0000 - tn0.5: 66082352.0000 - fn0.5: 3563132.0000 - precision0.5: 0.7416 - recall0.5: 0.6519 -
     tp0.7: 5275458.0000 - fp0.7: 916716.0000 - tn0.7: 67490392.0000 - fn0.7: 4960605.0000 - precision0.7: 0.8520 - recall0.7: 0.5154 - tp0.9: 3246371.0000 - fp0.9: 199829.0000 - tn0.9: 68207312.0000 - fn0.9: 6989692.0000 - precision0.9: 0.
    9420 - recall0.9: 0.3172 - accuracy: 0.9251 - auc: 0.9067 - f1: 0.6939 - val_loss: 1.4857 - val_tp0.1: 978503.0000 - val_fp0.1: 435953.0000 - val_tn0.1: 16502352.0000 - val_fn0.1: 1743992.0000 - val_precision0.1: 0.6918 - val_recall0.1:
     0.3594 - val_tp0.3: 873592.0000 - val_fp0.3: 222905.0000 - val_tn0.3: 16715400.0000 - val_fn0.3: 1848903.0000 - val_precision0.3: 0.7967 - val_recall0.3: 0.3209 - val_tp0.5: 816820.0000 - val_fp0.5: 140072.0000 - val_tn0.5: 16798232.00
    00 - val_fn0.5: 1905675.0000 - val_precision0.5: 0.8536 - val_recall0.5: 0.3000 - val_tp0.7: 747818.0000 - val_fp0.7: 84156.0000 - val_tn0.7: 16854148.0000 - val_fn0.7: 1974677.0000 - val_precision0.7: 0.8988 - val_recall0.7: 0.2747 - v
    al_tp0.9: 619063.0000 - val_fp0.9: 32598.0000 - val_tn0.9: 16905708.0000 - val_fn0.9: 2103432.0000 - val_precision0.9: 0.9500 - val_recall0.9: 0.2274 - val_accuracy: 0.8959 - val_auc: 0.7267 - val_f1: 0.4440
    
    ...
    
    Epoch 100/100
    240/240 [==============================] - 197s 821ms/step - loss: 0.0929 - tp0.1: 10067403.0000 - fp0.1: 1713493.0000 - tn0.1: 66693648.0000 - fn0.1: 168660.0000 - precision0.1: 0.8546 - recall0.1: 0.9835 - tp0.3: 9949382.0000 - fp0.3:
     1087364.0000 - tn0.3: 67319792.0000 - fn0.3: 286681.0000 - precision0.3: 0.9015 - recall0.3: 0.9720 - tp0.5: 9760046.0000 - fp0.5: 690436.0000 - tn0.5: 67716696.0000 - fn0.5: 476017.0000 - precision0.5: 0.9339 - recall0.5: 0.9535 - tp0
    .7: 9431582.0000 - fp0.7: 374056.0000 - tn0.7: 68033088.0000 - fn0.7: 804481.0000 - precision0.7: 0.9619 - recall0.7: 0.9214 - tp0.9: 8660415.0000 - fp0.9: 112381.0000 - tn0.9: 68294768.0000 - fn0.9: 1575648.0000 - precision0.9: 0.9872
    - recall0.9: 0.8461 - accuracy: 0.9852 - auc: 0.9906 - f1: 0.9436 - val_loss: 0.2611 - val_tp0.1: 2521251.0000 - val_fp0.1: 602159.0000 - val_tn0.1: 16336146.0000 - val_fn0.1: 201244.0000 - val_precision0.1: 0.8072 - val_recall0.1: 0.92
    61 - val_tp0.3: 2463263.0000 - val_fp0.3: 387212.0000 - val_tn0.3: 16551093.0000 - val_fn0.3: 259232.0000 - val_precision0.3: 0.8642 - val_recall0.3: 0.9048 - val_tp0.5: 2406415.0000 - val_fp0.5: 272305.0000 - val_tn0.5: 16666000.0000 -
     val_fn0.5: 316080.0000 - val_precision0.5: 0.8983 - val_recall0.5: 0.8839 - val_tp0.7: 2314277.0000 - val_fp0.7: 168490.0000 - val_tn0.7: 16769815.0000 - val_fn0.7: 408218.0000 - val_precision0.7: 0.9321 - val_recall0.7: 0.8501 - val_t
    p0.9: 2130323.0000 - val_fp0.9: 72929.0000 - val_tn0.9: 16865376.0000 - val_fn0.9: 592172.0000 - val_precision0.9: 0.9669 - val_recall0.9: 0.7825 - val_accuracy: 0.9701 - val_auc: 0.9595 - val_f1: 0.8911
    2022-03-01 18:11:37.935237: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
    2022/03/01 18:11:48 INFO mlflow.projects: === Run (ID '19e3e786e1bc4e2b93856f5dc9de8216') succeeded ===
    
    
    • name of run: 19e3e786e1bc4e2b93856f5dc9de8216
    • metrics after 100th epoch:
      • precisionval,0.5: 0.8983 - recallval,0.5: 0.8839 - f1val,0.5: 0.8911
      • aucval: 0.9595
  6. run training of (7cafab027cdd4fc9bf20a43e989df510, 16dff15d935f45e2a836b1f41b07b4e3) with hparams from above
    mlflow run . -e main -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ \
           -P batch_size=10 \
           -P first_filters=16 \
           -P input_size=14000 \
           -P lr_start=0.0627676336651573 \
           -P lr_power=1 \
           -P epochs=100 \
           -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets \
           -P csv_path_val=/beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN \
           -P scaler=robust \
           -P n_levels=5 \
           -P pool_size=4
    
    2022/03/01 19:36:22 INFO mlflow.projects.utils: === Created directory /tmp/tmpzbcp4g1j for downloading remote URIs passed to arguments of type 'path' ===
    2022/03/01 19:36:22 INFO mlflow.projects.backend.local: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe 1>&2 && python src/fluotracify/train
    ing/train.py --fluotracify_path /beegfs/ye53nis/drmed-git/src --batch_size 10 --input_size 14000 --lr_start 0.0627676336651573 --lr_power 1 --epochs 100 --csv_path_train /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets --csv_p
    ath_val /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN --col_per_example 3 --scaler robust --n_levels 5 --first_filters 16 --pool_size 4' in run with ID '347669d050f344ad9fb9e480c814f727' ===
    2022-03-01 19:36:34.348943: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-01 19:36:46,433 - train -  Python version: 3.9.6 (default, Jul 30 2021, 16:35:19)
    [GCC 7.5.0]
    2022-03-01 19:36:46,434 - train -  Tensorflow version: 2.5.0
    2022-03-01 19:36:46,434 - train -  tf.keras version: 2.5.0
    2022-03-01 19:36:46,434 - train -  Cudnn version: 8
    2022-03-01 19:36:46,434 - train -  Cuda version: 11.2
    2022-03-01 19:36:46.437417: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
    2022-03-01 19:36:46.506267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-03-01 19:36:46.506409: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-01 19:36:46.516098: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-03-01 19:36:46.516211: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    2022-03-01 19:36:46.519627: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
    2022-03-01 19:36:46.522117: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
    2022-03-01 19:36:46.531350: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
    2022-03-01 19:36:46.534380: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
    2022-03-01 19:36:46.536313: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-03-01 19:36:46.539533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-03-01 19:36:46,539 - train -  GPUs: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]. Trying to set memory growth to "True"...
    2022-03-01 19:36:46,540 - train -  Setting memory growth successful.
    2022-03-01 19:36:53,495 - train -  1/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set004.csv
    2022-03-01 19:36:59,518 - train -  2/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set002.csv
    2022-03-01 19:37:03,152 - train -  3/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set001.csv
    2022-03-01 19:37:06,774 - train -  4/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set007.csv
    2022-03-01 19:37:10,319 - train -  5/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
    2022-03-01 19:37:16,051 - train -  6/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/1.0/traces_brightclust_Nov2020_D10_set010.csv
    2022-03-01 19:37:20,347 - train -  7/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set004.csv
    2022-03-01 19:37:28,839 - train -  8/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.1/traces_brightclust_Nov2020_D3.0_set010.csv
    2022-03-01 19:37:32,240 - train -  9/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set009.csv
    2022-03-01 19:37:36,494 - train -  10/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set003.csv
    2022-03-01 19:37:40,435 - train -  11/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set007.csv
    2022-03-01 19:37:45,646 - train -  12/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
    2022-03-01 19:37:49,090 - train -  13/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set003.csv
    2022-03-01 19:37:53,145 - train -  14/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set002.csv
    2022-03-01 19:37:56,818 - train -  15/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set005.csv
    2022-03-01 19:38:00,166 - train -  16/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/1.0/traces_brightclust_Nov2020_D0.069_set009.csv
    2022-03-01 19:38:03,643 - train -  17/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set004.csv
    2022-03-01 19:38:07,281 - train -  18/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set010.csv
    2022-03-01 19:38:10,928 - train -  19/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set009.csv
    2022-03-01 19:38:14,614 - train -  20/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.01/traces_brightclust_Nov2020_D1.0_set010.csv
    2022-03-01 19:38:18,136 - train -  21/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set007.csv
    2022-03-01 19:38:21,832 - train -  22/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set010.csv
    2022-03-01 19:38:27,731 - train -  23/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set001.csv
    2022-03-01 19:38:31,406 - train -  24/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set003.csv
    2022-03-01 19:38:34,819 - train -  25/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set003.csv
    2022-03-01 19:38:38,249 - train -  26/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set004.csv
    2022-03-01 19:38:41,695 - train -  27/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set001.csv
    2022-03-01 19:38:45,025 - train -  28/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set005.csv
    2022-03-01 19:38:48,758 - train -  29/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set006.csv
    2022-03-01 19:38:52,913 - train -  30/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set004.csv
    2022-03-01 19:38:56,358 - train -  31/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set006.csv
    2022-03-01 19:39:00,987 - train -  32/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set004.csv
    2022-03-01 19:39:04,634 - train -  33/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set005.csv
    2022-03-01 19:39:08,400 - train -  34/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set003.csv
    2022-03-01 19:39:11,867 - train -  35/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.01/traces_brightclust_Nov2020_D0.2_set003.csv
    2022-03-01 19:39:15,951 - train -  36/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.01/traces_brightclust_Nov2020_D50_set006.csv
    2022-03-01 19:39:19,638 - train -  37/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set004.csv
    2022-03-01 19:39:23,207 - train -  38/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set004.csv
    2022-03-01 19:39:30,739 - train -  39/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.01/traces_brightclust_Nov2020_D0.6_set010.csv
    2022-03-01 19:39:34,471 - train -  40/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set007.csv
    2022-03-01 19:39:38,619 - train -  41/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set006.csv
    2022-03-01 19:39:42,300 - train -  42/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set002.csv
    2022-03-01 19:39:45,973 - train -  43/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set006.csv
    2022-03-01 19:39:49,635 - train -  44/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set006.csv
    2022-03-01 19:39:53,835 - train -  45/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set010.csv
    2022-03-01 19:39:57,292 - train -  46/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.01/traces_brightclust_Nov2020_D0.08_set007.csv
    2022-03-01 19:40:01,084 - train -  47/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set002.csv
    2022-03-01 19:40:04,949 - train -  48/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set009.csv
    2022-03-01 19:40:08,522 - train -  1/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/0.1/traces_brightclust_Nov2020_D1.0_set009.csv
    2022-03-01 19:40:11,907 - train -  2/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/50/1.0/traces_brightclust_Nov2020_D50_set007.csv
    2022-03-01 19:40:15,721 - train -  3/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.4/0.1/traces_brightclust_Nov2020_D0.4_set009.csv
    2022-03-01 19:40:19,221 - train -  4/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/1.0/traces_brightclust_Nov2020_D3.0_set009.csv
    2022-03-01 19:40:22,982 - train -  5/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.1/0.01/traces_brightclust_Nov2020_D0.1_set008.csv
    2022-03-01 19:40:26,520 - train -  6/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/10/0.01/traces_brightclust_Nov2020_D10_set008.csv
    2022-03-01 19:40:30,083 - train -  7/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.08/0.1/traces_brightclust_Nov2020_D0.08_set008.csv
    2022-03-01 19:40:33,515 - train -  8/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
    2022-03-01 19:40:36,862 - train -  9/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/0.01/traces_brightclust_Nov2020_D3.0_set008.csv
    2022-03-01 19:40:40,692 - train -  10/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.2/0.1/traces_brightclust_Nov2020_D0.2_set006.csv
    2022-03-01 19:40:44,478 - train -  11/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.069/0.1/traces_brightclust_Nov2020_D0.069_set006.csv
    2022-03-01 19:40:47,736 - train -  12/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.6/0.1/traces_brightclust_Nov2020_D0.6_set006.csv
    2022-03-01 19:40:47,939 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 4800), (16384, 4800), (16384, 4800)]
    2022-03-01 19:40:48,032 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 1200), (16384, 1200), (16384, 1200)]
    2022-03-01 19:40:48.195482: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operatio
    ns:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    2022-03-01 19:40:48.198562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-03-01 19:40:48.200594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-03-01 19:40:48.200695: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-01 19:40:48.635293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
    2022-03-01 19:40:48.635366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0
    2022-03-01 19:40:48.635382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N
    2022-03-01 19:40:48.638373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15403 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, p
    ci bus id: 0000:82:00.0, compute capability: 6.0)
    2022-03-01 19:40:51,863 - train -  number of examples: 4800
    2022-03-01 19:40:52,254 - train -  number of examples: 1200
    2022-03-01 19:40:54,716 - train -  unet: input shape: (None, None, 1), output shape: (None, None, 1)
    2022/03/01 19:40:54 WARNING mlflow.utils.autologging_utils: Encountered unexpected error during tensorflow autologging: Changing param values is not allowed. Param with key='batch_size' was already logged with value='10' for run ID='347
    669d050f344ad9fb9e480c814f727'. Attempted logging new value 'None'.
    Epoch 1/100
    2022-03-01 19:41:04.464586: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
    2022-03-01 19:41:04.627492: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199895000 Hz
    2022-03-01 19:41:14.720330: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-03-01 19:41:15.055044: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101
    2022-03-01 19:41:15.546433: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-03-01 19:41:15.832893: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    480/480 [==============================] - 60s 82ms/step - loss: 0.5773 - tp0.1: 8880905.0000 - fp0.1: 10094803.0000 - tn0.1: 58312352.0000 - fn0.1: 1355158.0000 - precision0.1: 0.4680 - recall0.1: 0.8676 - tp0.3: 7871045.0000 - fp0.3:
    4514511.0000 - tn0.3: 63892572.0000 - fn0.3: 2365018.0000 - precision0.3: 0.6355 - recall0.3: 0.7690 - tp0.5: 6608273.0000 - fp0.5: 1693677.0000 - tn0.5: 66713460.0000 - fn0.5: 3627790.0000 - precision0.5: 0.7960 - recall0.5: 0.6456 - t
    p0.7: 5575527.0000 - fp0.7: 626556.0000 - tn0.7: 67780600.0000 - fn0.7: 4660536.0000 - precision0.7: 0.8990 - recall0.7: 0.5447 - tp0.9: 4341516.0000 - fp0.9: 163073.0000 - tn0.9: 68244088.0000 - fn0.9: 5894547.0000 - precision0.9: 0.96
    38 - recall0.9: 0.4241 - accuracy: 0.9323 - auc: 0.9124 - f1: 0.7129 - val_loss: 0.6884 - val_tp0.1: 2348236.0000 - val_fp0.1: 3299237.0000 - val_tn0.1: 13639068.0000 - val_fn0.1: 374259.0000 - val_precision0.1: 0.4158 - val_recall0.1:
    0.8625 - val_tp0.3: 2180062.0000 - val_fp0.3: 2146157.0000 - val_tn0.3: 14792148.0000 - val_fn0.3: 542433.0000 - val_precision0.3: 0.5039 - val_recall0.3: 0.8008 - val_tp0.5: 1966718.0000 - val_fp0.5: 1074503.0000 - val_tn0.5: 15863802.
    0000 - val_fn0.5: 755777.0000 - val_precision0.5: 0.6467 - val_recall0.5: 0.7224 - val_tp0.7: 1647155.0000 - val_fp0.7: 270297.0000 - val_tn0.7: 16668008.0000 - val_fn0.7: 1075340.0000 - val_precision0.7: 0.8590 - val_recall0.7: 0.6050
    - val_tp0.9: 1346781.0000 - val_fp0.9: 48532.0000 - val_tn0.9: 16889772.0000 - val_fn0.9: 1375714.0000 - val_precision0.9: 0.9652 - val_recall0.9: 0.4947 - val_accuracy: 0.9069 - val_auc: 0.8983 - val_f1: 0.6824
    
    ...
    
    Epoch 100/100
    480/480 [==============================] - 36s 75ms/step - loss: 0.0973 - tp0.1: 10083866.0000 - fp0.1: 2150370.0000 - tn0.1: 66256776.0000 - fn0.1: 152197.0000 - precision0.1: 0.8242 - recall0.1: 0.9851 - tp0.3: 9911194.0000 - fp0.3: 1
    217280.0000 - tn0.3: 67189872.0000 - fn0.3: 324869.0000 - precision0.3: 0.8906 - recall0.3: 0.9683 - tp0.5: 9675199.0000 - fp0.5: 716243.0000 - tn0.5: 67690904.0000 - fn0.5: 560864.0000 - precision0.5: 0.9311 - recall0.5: 0.9452 - tp0.7
    : 9261235.0000 - fp0.7: 330050.0000 - tn0.7: 68077096.0000 - fn0.7: 974828.0000 - precision0.7: 0.9656 - recall0.7: 0.9048 - tp0.9: 8585163.0000 - fp0.9: 94613.0000 - tn0.9: 68312512.0000 - fn0.9: 1650900.0000 - precision0.9: 0.9891 - r
    ecall0.9: 0.8387 - accuracy: 0.9838 - auc: 0.9915 - f1: 0.9381 - val_loss: 0.1400 - val_tp0.1: 2648521.0000 - val_fp0.1: 631749.0000 - val_tn0.1: 16306556.0000 - val_fn0.1: 73974.0000 - val_precision0.1: 0.8074 - val_recall0.1: 0.9728 -
     val_tp0.3: 2586061.0000 - val_fp0.3: 344848.0000 - val_tn0.3: 16593457.0000 - val_fn0.3: 136434.0000 - val_precision0.3: 0.8823 - val_recall0.3: 0.9499 - val_tp0.5: 2514973.0000 - val_fp0.5: 202882.0000 - val_tn0.5: 16735423.0000 - val
    _fn0.5: 207522.0000 - val_precision0.5: 0.9254 - val_recall0.5: 0.9238 - val_tp0.7: 2409626.0000 - val_fp0.7: 97831.0000 - val_tn0.7: 16840474.0000 - val_fn0.7: 312869.0000 - val_precision0.7: 0.9610 - val_recall0.7: 0.8851 - val_tp0.9:
     2228029.0000 - val_fp0.9: 30092.0000 - val_tn0.9: 16908212.0000 - val_fn0.9: 494466.0000 - val_precision0.9: 0.9867 - val_recall0.9: 0.8184 - val_accuracy: 0.9791 - val_auc: 0.9848 - val_f1: 0.9246
    2022-03-01 20:46:33.038541: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
    2022/03/01 20:46:47 INFO mlflow.projects: === Run (ID '347669d050f344ad9fb9e480c814f727') succeeded ===
    
    
    • name of run: 347669d050f344ad9fb9e480c814f727
    • metrics after 100th epoch:
      • precisionval,0.5: 0.9254 - recallval,0.5: 0.9238 - f1val,0.5: 0.9246
      • aucval: 0.9848
  7. run training of (0e328920e86049928202db95e8cfb7be, bf9d2725eb16462d9a101f0a077ce2b5) with hparams from above
    mlflow run . -e main -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ \
           -P batch_size=14 \
           -P first_filters=16 \
           -P input_size=14000 \
           -P lr_start=0.0192390310290551 \
           -P lr_power=5 \
           -P epochs=100 \
           -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets \
           -P csv_path_val=/beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN \
           -P scaler=robust \
           -P n_levels=9 \
           -P pool_size=2
    
    2022/03/01 22:12:06 INFO mlflow.projects.utils: === Created directory /tmp/tmpgcmidltj for downloading remote URIs passed to arguments of type 'path' ===
    2022/03/01 22:12:06 INFO mlflow.projects.backend.local: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe 1>&2 && python src/fluotracify/train
    ing/train.py --fluotracify_path /beegfs/ye53nis/drmed-git/src --batch_size 14 --input_size 14000 --lr_start 0.0192390310290551 --lr_power 5 --epochs 100 --csv_path_train /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets --csv_p
    ath_val /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN --col_per_example 3 --scaler robust --n_levels 9 --first_filters 16 --pool_size 2' in run with ID 'c1204e3a8a1e4c40a35b5b7b1922d1ce' ===
    2022-03-01 22:12:20.335142: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-01 22:12:33,450 - train -  Python version: 3.9.6 (default, Jul 30 2021, 16:35:19)
    [GCC 7.5.0]
    2022-03-01 22:12:33,451 - train -  Tensorflow version: 2.5.0
    2022-03-01 22:12:33,451 - train -  tf.keras version: 2.5.0
    2022-03-01 22:12:33,451 - train -  Cudnn version: 8
    2022-03-01 22:12:33,451 - train -  Cuda version: 11.2
    2022-03-01 22:12:33.453876: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
    2022-03-01 22:12:33.508402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-03-01 22:12:33.508543: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-01 22:12:33.518297: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-03-01 22:12:33.518392: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    2022-03-01 22:12:33.521775: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
    2022-03-01 22:12:33.523563: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
    2022-03-01 22:12:33.532074: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
    2022-03-01 22:12:33.534870: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
    2022-03-01 22:12:33.536502: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-03-01 22:12:33.539631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-03-01 22:12:33,539 - train -  GPUs: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]. Trying to set memory growth to "True"...
    2022-03-01 22:12:33,540 - train -  Setting memory growth successful.
    2022-03-01 22:12:39,562 - train -  1/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set004.csv
    2022-03-01 22:12:42,849 - train -  2/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set002.csv
    2022-03-01 22:12:46,207 - train -  3/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set001.csv
    2022-03-01 22:12:50,855 - train -  4/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set007.csv
    2022-03-01 22:12:55,527 - train -  5/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
    2022-03-01 22:12:58,713 - train -  6/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/1.0/traces_brightclust_Nov2020_D10_set010.csv
    2022-03-01 22:13:02,325 - train -  7/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set004.csv
    2022-03-01 22:13:05,461 - train -  8/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.1/traces_brightclust_Nov2020_D3.0_set010.csv
    2022-03-01 22:13:08,927 - train -  9/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set009.csv
    2022-03-01 22:13:14,609 - train -  10/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set003.csv
    2022-03-01 22:13:17,983 - train -  11/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set007.csv
    2022-03-01 22:13:21,031 - train -  12/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
    2022-03-01 22:13:24,219 - train -  13/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set003.csv
    2022-03-01 22:13:27,321 - train -  14/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set002.csv
    2022-03-01 22:13:30,784 - train -  15/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set005.csv
    2022-03-01 22:13:34,829 - train -  16/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/1.0/traces_brightclust_Nov2020_D0.069_set009.csv
    2022-03-01 22:13:38,078 - train -  17/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set004.csv
    2022-03-01 22:13:41,302 - train -  18/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set010.csv
    2022-03-01 22:13:45,845 - train -  19/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set009.csv
    2022-03-01 22:13:49,546 - train -  20/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.01/traces_brightclust_Nov2020_D1.0_set010.csv
    2022-03-01 22:13:53,591 - train -  21/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set007.csv
    2022-03-01 22:13:56,776 - train -  22/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set010.csv
    2022-03-01 22:14:00,326 - train -  23/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set001.csv
    2022-03-01 22:14:03,666 - train -  24/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set003.csv
    2022-03-01 22:14:06,840 - train -  25/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set003.csv
    2022-03-01 22:14:12,440 - train -  26/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set004.csv
    2022-03-01 22:14:15,561 - train -  27/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set001.csv
    2022-03-01 22:14:18,839 - train -  28/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set005.csv
    2022-03-01 22:14:22,240 - train -  29/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set006.csv
    2022-03-01 22:14:25,355 - train -  30/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set004.csv
    2022-03-01 22:14:28,483 - train -  31/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set006.csv
    2022-03-01 22:14:31,960 - train -  32/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set004.csv
    2022-03-01 22:14:35,186 - train -  33/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set005.csv
    2022-03-01 22:14:38,725 - train -  34/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set003.csv
    2022-03-01 22:14:42,022 - train -  35/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.01/traces_brightclust_Nov2020_D0.2_set003.csv
    2022-03-01 22:14:45,344 - train -  36/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.01/traces_brightclust_Nov2020_D50_set006.csv
    2022-03-01 22:14:48,717 - train -  37/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set004.csv
    2022-03-01 22:14:52,459 - train -  38/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set004.csv
    2022-03-01 22:14:57,077 - train -  39/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.01/traces_brightclust_Nov2020_D0.6_set010.csv
    2022-03-01 22:15:00,330 - train -  40/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set007.csv
    2022-03-01 22:15:03,675 - train -  41/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set006.csv
    2022-03-01 22:15:07,194 - train -  42/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set002.csv
    2022-03-01 22:15:10,373 - train -  43/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set006.csv
    2022-03-01 22:15:13,913 - train -  44/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set006.csv
    2022-03-01 22:15:17,100 - train -  45/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set010.csv
    2022-03-01 22:15:20,452 - train -  46/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.01/traces_brightclust_Nov2020_D0.08_set007.csv
    2022-03-01 22:15:23,646 - train -  47/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set002.csv
    2022-03-01 22:15:27,210 - train -  48/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set009.csv
    2022-03-01 22:15:31,036 - train -  1/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/0.1/traces_brightclust_Nov2020_D1.0_set009.csv
    2022-03-01 22:15:35,097 - train -  2/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/50/1.0/traces_brightclust_Nov2020_D50_set007.csv
    2022-03-01 22:15:38,379 - train -  3/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.4/0.1/traces_brightclust_Nov2020_D0.4_set009.csv
    2022-03-01 22:15:41,484 - train -  4/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/1.0/traces_brightclust_Nov2020_D3.0_set009.csv
    2022-03-01 22:15:44,853 - train -  5/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.1/0.01/traces_brightclust_Nov2020_D0.1_set008.csv
    2022-03-01 22:15:50,250 - train -  6/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/10/0.01/traces_brightclust_Nov2020_D10_set008.csv
    2022-03-01 22:15:54,918 - train -  7/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.08/0.1/traces_brightclust_Nov2020_D0.08_set008.csv
    2022-03-01 22:15:58,298 - train -  8/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
    2022-03-01 22:16:02,901 - train -  9/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/0.01/traces_brightclust_Nov2020_D3.0_set008.csv
    2022-03-01 22:16:06,154 - train -  10/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.2/0.1/traces_brightclust_Nov2020_D0.2_set006.csv
    2022-03-01 22:16:09,817 - train -  11/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.069/0.1/traces_brightclust_Nov2020_D0.069_set006.csv
    2022-03-01 22:16:12,969 - train -  12/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.6/0.1/traces_brightclust_Nov2020_D0.6_set006.csv
    2022-03-01 22:16:13,193 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 4800), (16384, 4800), (16384, 4800)]
    2022-03-01 22:16:13,282 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 1200), (16384, 1200), (16384, 1200)]
    2022-03-01 22:16:13.549849: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operatio
    ns:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    2022-03-01 22:16:13.552236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-03-01 22:16:13.554196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-03-01 22:16:13.554316: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-01 22:16:13.976589: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
    2022-03-01 22:16:13.976661: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0
    2022-03-01 22:16:13.976674: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N
    2022-03-01 22:16:13.979482: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15403 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, p
    ci bus id: 0000:82:00.0, compute capability: 6.0)
    2022-03-01 22:16:15,961 - train -  number of examples: 4800
    2022-03-01 22:16:16,380 - train -  number of examples: 1200
    2022-03-01 22:16:19,293 - train -  unet: input shape: (None, None, 1), output shape: (None, None, 1)
    2022/03/01 22:16:19 WARNING mlflow.utils.autologging_utils: Encountered unexpected error during tensorflow autologging: Changing param values is not allowed. Param with key='batch_size' was already logged with value='14' for run ID='c12
    04e3a8a1e4c40a35b5b7b1922d1ce'. Attempted logging new value 'None'.
    Epoch 1/100
    2022-03-01 22:16:30.776559: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
    2022-03-01 22:16:30.983303: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199895000 Hz
    2022-03-01 22:16:42.127026: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-03-01 22:16:42.439526: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101
    2022-03-01 22:16:42.928881: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-03-01 22:16:43.210835: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    342/342 [==============================] - 124s 292ms/step - loss: 0.7070 - tp0.1: 8568535.0000 - fp0.1: 13083853.0000 - tn0.1: 55158168.0000 - fn0.1: 1636041.0000 - precision0.1: 0.3957 - recall0.1: 0.8397 - tp0.3: 6954119.0000 - fp0.3
    : 4121236.0000 - tn0.3: 64120780.0000 - fn0.3: 3250457.0000 - precision0.3: 0.6279 - recall0.3: 0.6815 - tp0.5: 5914618.0000 - fp0.5: 1373295.0000 - tn0.5: 66868700.0000 - fn0.5: 4289958.0000 - precision0.5: 0.8116 - recall0.5: 0.5796 -
     tp0.7: 5157583.0000 - fp0.7: 495656.0000 - tn0.7: 67746344.0000 - fn0.7: 5046993.0000 - precision0.7: 0.9123 - recall0.7: 0.5054 - tp0.9: 4126657.0000 - fp0.9: 91030.0000 - tn0.9: 68151000.0000 - fn0.9: 6077919.0000 - precision0.9: 0.9
    784 - recall0.9: 0.4044 - accuracy: 0.9278 - auc: 0.8844 - f1: 0.6762 - val_loss: 0.6968 - val_tp0.1: 2338956.0000 - val_fp0.1: 2679424.0000 - val_tn0.1: 14116949.0000 - val_fn0.1: 361631.0000 - val_precision0.1: 0.4661 - val_recall0.1:
     0.8661 - val_tp0.3: 2156286.0000 - val_fp0.3: 1498035.0000 - val_tn0.3: 15298338.0000 - val_fn0.3: 544301.0000 - val_precision0.3: 0.5901 - val_recall0.3: 0.7985 - val_tp0.5: 1886303.0000 - val_fp0.5: 688918.0000 - val_tn0.5: 16107455.
    0000 - val_fn0.5: 814284.0000 - val_precision0.5: 0.7325 - val_recall0.5: 0.6985 - val_tp0.7: 1720047.0000 - val_fp0.7: 358918.0000 - val_tn0.7: 16437455.0000 - val_fn0.7: 980540.0000 - val_precision0.7: 0.8274 - val_recall0.7: 0.6369 -
     val_tp0.9: 1536712.0000 - val_fp0.9: 146022.0000 - val_tn0.9: 16650351.0000 - val_fn0.9: 1163875.0000 - val_precision0.9: 0.9132 - val_recall0.9: 0.5690 - val_accuracy: 0.9229 - val_auc: 0.9057 - val_f1: 0.7151
    
    ...
    
    Epoch 100/100
    342/342 [==============================] - 96s 282ms/step - loss: 0.1201 - tp0.1: 9816720.0000 - fp0.1: 2613299.0000 - tn0.1: 65809824.0000 - fn0.1: 206742.0000 - precision0.1: 0.7898 - recall0.1: 0.9794 - tp0.3: 9593189.0000 - fp0.3: 1
    365697.0000 - tn0.3: 67057408.0000 - fn0.3: 430273.0000 - precision0.3: 0.8754 - recall0.3: 0.9571 - tp0.5: 9310282.0000 - fp0.5: 760248.0000 - tn0.5: 67662912.0000 - fn0.5: 713180.0000 - precision0.5: 0.9245 - recall0.5: 0.9288 - tp0.7
    : 8896402.0000 - fp0.7: 359466.0000 - tn0.7: 68063680.0000 - fn0.7: 1127060.0000 - precision0.7: 0.9612 - recall0.7: 0.8876 - tp0.9: 8125429.0000 - fp0.9: 93565.0000 - tn0.9: 68329576.0000 - fn0.9: 1898033.0000 - precision0.9: 0.9886 -
    recall0.9: 0.8106 - accuracy: 0.9812 - auc: 0.9882 - f1: 0.9267 - val_loss: 0.1429 - val_tp0.1: 2631773.0000 - val_fp0.1: 731922.0000 - val_tn0.1: 16067311.0000 - val_fn0.1: 65954.0000 - val_precision0.1: 0.7824 - val_recall0.1: 0.9756
    - val_tp0.3: 2571221.0000 - val_fp0.3: 390541.0000 - val_tn0.3: 16408692.0000 - val_fn0.3: 126506.0000 - val_precision0.3: 0.8681 - val_recall0.3: 0.9531 - val_tp0.5: 2491171.0000 - val_fp0.5: 222824.0000 - val_tn0.5: 16576409.0000 - va
    l_fn0.5: 206556.0000 - val_precision0.5: 0.9179 - val_recall0.5: 0.9234 - val_tp0.7: 2379767.0000 - val_fp0.7: 112408.0000 - val_tn0.7: 16686825.0000 - val_fn0.7: 317960.0000 - val_precision0.7: 0.9549 - val_recall0.7: 0.8821 - val_tp0.
    9: 2166171.0000 - val_fp0.9: 33767.0000 - val_tn0.9: 16765466.0000 - val_fn0.9: 531556.0000 - val_precision0.9: 0.9847 - val_recall0.9: 0.8030 - val_accuracy: 0.9780 -
    2022-03-02 01:02:37.643121: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
    2022/03/02 01:03:00 INFO mlflow.projects: === Run (ID 'c1204e3a8a1e4c40a35b5b7b1922d1ce') succeeded ===
    (tf) [ye53nis@node128 drmed-git]$
    
    • name of run: c1204e3a8a1e4c40a35b5b7b1922d1ce
    • metrics after 100th epoch:
      • precisionval,0.5: 0.9179 - recallval,0.5: 0.9234 - f1val,0.5: 0.9207
      • aucval: 0.9858
  8. run training of (3cbd945b62ec4634839372e403f6f377, 458b36a70db843719d202a8eda448f17) with hparams from above
    mlflow run . -e main -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ \
           -P batch_size=9 \
           -P first_filters=64 \
           -P input_size=14000 \
           -P lr_start=0.0100697459464075 \
           -P lr_power=1 \
           -P epochs=100 \
           -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets \
           -P csv_path_val=/beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN \
           -P scaler=maxabs \
           -P n_levels=5 \
           -P pool_size=4
    
    2022/03/02 01:11:44 INFO mlflow.projects.utils: === Created directory /tmp/tmpx4epxfnm for downloading remote URIs passed to arguments of type 'path' ===
    2022/03/02 01:11:44 INFO mlflow.projects.backend.local: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe 1>&2 && python src/fluotracify/train
    ing/train.py --fluotracify_path /beegfs/ye53nis/drmed-git/src --batch_size 9 --input_size 14000 --lr_start 0.0100697459464075 --lr_power 1 --epochs 100 --csv_path_train /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets --csv_pa
    th_val /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN --col_per_example 3 --scaler maxabs --n_levels 5 --first_filters 64 --pool_size 4' in run with ID '714af8cd12c1441eac4ca980e8c20070' ===
    2022-03-02 01:11:56.803319: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-02 01:12:07,546 - train -  Python version: 3.9.6 (default, Jul 30 2021, 16:35:19)
    [GCC 7.5.0]
    2022-03-02 01:12:07,546 - train -  Tensorflow version: 2.5.0
    2022-03-02 01:12:07,546 - train -  tf.keras version: 2.5.0
    2022-03-02 01:12:07,547 - train -  Cudnn version: 8
    2022-03-02 01:12:07,547 - train -  Cuda version: 11.2
    2022-03-02 01:12:07.550455: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
    2022-03-02 01:12:07.628356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-03-02 01:12:07.628511: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-02 01:12:07.638859: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-03-02 01:12:07.638985: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    2022-03-02 01:12:07.643056: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
    2022-03-02 01:12:07.646183: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
    2022-03-02 01:12:07.656023: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
    2022-03-02 01:12:07.659075: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
    2022-03-02 01:12:07.661143: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-03-02 01:12:07.664383: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-03-02 01:12:07,664 - train -  GPUs: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]. Trying to set memory growth to "True"...
    2022-03-02 01:12:07,665 - train -  Setting memory growth successful.
    2022-03-02 01:12:13,856 - train -  1/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set004.csv
    2022-03-02 01:12:17,659 - train -  2/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set002.csv
    2022-03-02 01:12:21,236 - train -  3/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set001.csv
    2022-03-02 01:12:24,720 - train -  4/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set007.csv
    2022-03-02 01:12:27,968 - train -  5/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
    2022-03-02 01:12:31,066 - train -  6/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/1.0/traces_brightclust_Nov2020_D10_set010.csv
    2022-03-02 01:12:34,437 - train -  7/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set004.csv
    2022-03-02 01:12:37,504 - train -  8/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.1/traces_brightclust_Nov2020_D3.0_set010.csv
    2022-03-02 01:12:42,706 - train -  9/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set009.csv
    2022-03-02 01:12:46,132 - train -  10/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set003.csv
    2022-03-02 01:12:49,635 - train -  11/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set007.csv
    2022-03-02 01:12:53,763 - train -  12/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
    2022-03-02 01:12:57,006 - train -  13/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set003.csv
    2022-03-02 01:13:00,289 - train -  14/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set002.csv
    2022-03-02 01:13:03,628 - train -  15/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set005.csv
    2022-03-02 01:13:07,750 - train -  16/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/1.0/traces_brightclust_Nov2020_D0.069_set009.csv
    2022-03-02 01:13:11,178 - train -  17/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set004.csv
    2022-03-02 01:13:14,491 - train -  18/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set010.csv
    2022-03-02 01:13:17,835 - train -  19/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set009.csv
    2022-03-02 01:13:21,243 - train -  20/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.01/traces_brightclust_Nov2020_D1.0_set010.csv
    2022-03-02 01:13:24,830 - train -  21/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set007.csv
    2022-03-02 01:13:28,194 - train -  22/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set010.csv
    2022-03-02 01:13:31,658 - train -  23/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set001.csv
    2022-03-02 01:13:35,112 - train -  24/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set003.csv
    2022-03-02 01:13:38,261 - train -  25/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set003.csv
    2022-03-02 01:13:41,634 - train -  26/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set004.csv
    2022-03-02 01:13:44,894 - train -  27/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set001.csv
    2022-03-02 01:13:48,172 - train -  28/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set005.csv
    2022-03-02 01:13:51,792 - train -  29/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set006.csv
    2022-03-02 01:13:55,890 - train -  30/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set004.csv
    2022-03-02 01:13:59,226 - train -  31/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set006.csv
    2022-03-02 01:14:03,473 - train -  32/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set004.csv
    2022-03-02 01:14:06,889 - train -  33/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set005.csv
    2022-03-02 01:14:11,194 - train -  34/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set003.csv
    2022-03-02 01:14:14,602 - train -  35/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.01/traces_brightclust_Nov2020_D0.2_set003.csv
    2022-03-02 01:14:18,195 - train -  36/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.01/traces_brightclust_Nov2020_D50_set006.csv
    2022-03-02 01:14:23,874 - train -  37/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set004.csv
    2022-03-02 01:14:27,349 - train -  38/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set004.csv
    2022-03-02 01:14:30,931 - train -  39/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.01/traces_brightclust_Nov2020_D0.6_set010.csv
    2022-03-02 01:14:39,147 - train -  40/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set007.csv
    2022-03-02 01:14:42,631 - train -  41/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set006.csv
    2022-03-02 01:14:46,112 - train -  42/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set002.csv
    2022-03-02 01:14:49,382 - train -  43/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set006.csv
    2022-03-02 01:14:54,801 - train -  44/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set006.csv
    2022-03-02 01:14:58,059 - train -  45/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set010.csv
    2022-03-02 01:15:01,699 - train -  46/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.01/traces_brightclust_Nov2020_D0.08_set007.csv
    2022-03-02 01:15:04,960 - train -  47/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set002.csv
    2022-03-02 01:15:08,548 - train -  48/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set009.csv
    2022-03-02 01:15:11,862 - train -  1/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/0.1/traces_brightclust_Nov2020_D1.0_set009.csv
    2022-03-02 01:15:15,127 - train -  2/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/50/1.0/traces_brightclust_Nov2020_D50_set007.csv
    2022-03-02 01:15:18,793 - train -  3/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.4/0.1/traces_brightclust_Nov2020_D0.4_set009.csv
    2022-03-02 01:15:24,397 - train -  4/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/1.0/traces_brightclust_Nov2020_D3.0_set009.csv
    2022-03-02 01:15:27,879 - train -  5/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.1/0.01/traces_brightclust_Nov2020_D0.1_set008.csv
    2022-03-02 01:15:31,270 - train -  6/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/10/0.01/traces_brightclust_Nov2020_D10_set008.csv
    2022-03-02 01:15:34,589 - train -  7/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.08/0.1/traces_brightclust_Nov2020_D0.08_set008.csv
    2022-03-02 01:15:38,546 - train -  8/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
    2022-03-02 01:15:42,044 - train -  9/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/0.01/traces_brightclust_Nov2020_D3.0_set008.csv
    2022-03-02 01:15:45,426 - train -  10/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.2/0.1/traces_brightclust_Nov2020_D0.2_set006.csv
    2022-03-02 01:15:49,033 - train -  11/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.069/0.1/traces_brightclust_Nov2020_D0.069_set006.csv
    2022-03-02 01:15:52,325 - train -  12/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.6/0.1/traces_brightclust_Nov2020_D0.6_set006.csv
    2022-03-02 01:15:52,527 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 4800), (16384, 4800), (16384, 4800)]
    2022-03-02 01:15:52,616 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 1200), (16384, 1200), (16384, 1200)]
    2022-03-02 01:15:52.773335: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operatio
    ns:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    2022-03-02 01:15:52.775655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-03-02 01:15:52.777670: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-03-02 01:15:52.777752: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-02 01:15:53.202763: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
    2022-03-02 01:15:53.202835: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0
    2022-03-02 01:15:53.202850: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N
    2022-03-02 01:15:53.205744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15403 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, p
    ci bus id: 0000:82:00.0, compute capability: 6.0)
    2022-03-02 01:15:55,214 - train -  number of examples: 4800
    2022-03-02 01:15:55,624 - train -  number of examples: 1200
    2022-03-02 01:15:57,731 - train -  unet: input shape: (None, None, 1), output shape: (None, None, 1)
    2022/03/02 01:15:57 WARNING mlflow.utils.autologging_utils: Encountered unexpected error during tensorflow autologging: Changing param values is not allowed. Param with key='batch_size' was already logged with value='9' for run ID='714a
    f8cd12c1441eac4ca980e8c20070'. Attempted logging new value 'None'.
    Epoch 1/100
    2022-03-02 01:16:06.204044: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
    2022-03-02 01:16:06.363399: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199895000 Hz
    2022-03-02 01:16:14.227408: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-03-02 01:16:14.534785: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101
    2022-03-02 01:16:15.007638: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-03-02 01:16:15.294778: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    533/533 [==============================] - 129s 209ms/step - loss: 0.9630 - tp0.1: 8826943.0000 - fp0.1: 17028400.0000 - tn0.1: 51333204.0000 - fn0.1: 1405497.0000 - precision0.1: 0.3414 - recall0.1: 0.8626 - tp0.3: 7038448.0000 - fp0.3
    : 7003701.0000 - tn0.3: 61357888.0000 - fn0.3: 3193992.0000 - precision0.3: 0.5012 - recall0.3: 0.6879 - tp0.5: 4980151.0000 - fp0.5: 2201947.0000 - tn0.5: 66159680.0000 - fn0.5: 5252289.0000 - precision0.5: 0.6934 - recall0.5: 0.4867 -
     tp0.7: 3822069.0000 - fp0.7: 917042.0000 - tn0.7: 67444552.0000 - fn0.7: 6410371.0000 - precision0.7: 0.8065 - recall0.7: 0.3735 - tp0.9: 2341770.0000 - fp0.9: 282920.0000 - tn0.9: 68078696.0000 - fn0.9: 7890670.0000 - precision0.9: 0.
    8922 - recall0.9: 0.2289 - accuracy: 0.9052 - auc: 0.8844 - f1: 0.5720 - val_loss: 17.7890 - val_tp0.1: 2710616.0000 - val_fp0.1: 15396795.0000 - val_tn0.1: 1498695.0000 - val_fn0.1: 5542.0000 - val_precision0.1: 0.1497 - val_recall0.1:
     0.9980 - val_tp0.3: 2709922.0000 - val_fp0.3: 15341678.0000 - val_tn0.3: 1553812.0000 - val_fn0.3: 6236.0000 - val_precision0.3: 0.1501 - val_recall0.3: 0.9977 - val_tp0.5: 2709375.0000 - val_fp0.5: 15293665.0000 - val_tn0.5: 1601825.0
    000 - val_fn0.5: 6783.0000 - val_precision0.5: 0.1505 - val_recall0.5: 0.9975 - val_tp0.7: 2708822.0000 - val_fp0.7: 15242876.0000 - val_tn0.7: 1652614.0000 - val_fn0.7: 7336.0000 - val_precision0.7: 0.1509 - val_recall0.7: 0.9973 - val
    _tp0.9: 2707532.0000 - val_fp0.9: 15151973.0000 - val_tn0.9: 1743517.0000 - val_fn0.9: 8626.0000 - val_precision0.9: 0.1516 - val_recall0.9: 0.9968 - val_accuracy: 0.2198 - val_auc: 0.5561 - val_f1: 0.2615
    
    ...
    
    Epoch 100/100
    533/533 [==============================] - 108s 202ms/step - loss: 0.0724 - tp0.1: 10196374.0000 - fp0.1: 1589491.0000 - tn0.1: 66701600.0000 - fn0.1: 106580.0000 - precision0.1: 0.8651 - recall0.1: 0.9897 - tp0.3: 10080835.0000 - fp0.3
    : 986714.0000 - tn0.3: 67304368.0000 - fn0.3: 222119.0000 - precision0.3: 0.9108 - recall0.3: 0.9784 - tp0.5: 9895952.0000 - fp0.5: 600682.0000 - tn0.5: 67690432.0000 - fn0.5: 407002.0000 - precision0.5: 0.9428 - recall0.5: 0.9605 - tp0
    .7: 9574784.0000 - fp0.7: 291406.0000 - tn0.7: 67999680.0000 - fn0.7: 728170.0000 - precision0.7: 0.9705 - recall0.7: 0.9293 - tp0.9: 8942531.0000 - fp0.9: 72576.0000 - tn0.9: 68218544.0000 - fn0.9: 1360423.0000 - precision0.9: 0.9919 -
     recall0.9: 0.8680 - accuracy: 0.9872 - auc: 0.9942 - f1: 0.9516 - val_loss: 0.1303 - val_tp0.1: 2631957.0000 - val_fp0.1: 478568.0000 - val_tn0.1: 16426044.0000 - val_fn0.1: 75079.0000 - val_precision0.1: 0.8461 - val_recall0.1: 0.9723
     - val_tp0.3: 2587183.0000 - val_fp0.3: 308683.0000 - val_tn0.3: 16595929.0000 - val_fn0.3: 119853.0000 - val_precision0.3: 0.8934 - val_recall0.3: 0.9557 - val_tp0.5: 2531594.0000 - val_fp0.5: 203214.0000 - val_tn0.5: 16701398.0000 - v
    al_fn0.5: 175442.0000 - val_precision0.5: 0.9257 - val_recall0.5: 0.9352 - val_tp0.7: 2448316.0000 - val_fp0.7: 118418.0000 - val_tn0.7: 16786194.0000 - val_fn0.7: 258720.0000 - val_precision0.7: 0.9539 - val_recall0.7: 0.9044 - val_tp0
    .9: 2283631.0000 - val_fp0.9: 49167.0000 - val_tn0.9: 16855444.0000 - val_fn0.9: 423405.0000 - val_precision0.9: 0.9789 - val_recall0.9: 0.8436 - val_accuracy: 0.9807 - val_auc: 0.9843 - val_f1: 0.9304
    2022-03-02 04:20:37.604740: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
    2022/03/02 04:20:52 INFO mlflow.projects: === Run (ID '714af8cd12c1441eac4ca980e8c20070') succeeded ===
    
    
    • name of run: 714af8cd12c1441eac4ca980e8c20070
    • metrics after 100th epoch:
      • loss: 0.0724 - lossval: 0.1303
      • precisionval,0.5: 0.9257 - recallval,0.5: 0.9352 - f1val,0.5: 0.9304
      • aucval: 0.9843
  9. run training of (3cbd945b62ec4634839372e403f6f377, 458b36a70db843719d202a8eda448f17) with hparams from above
    mlflow run . -e main -P fluotracify_path=/beegfs/ye53nis/drmed-git/src/ \
           -P batch_size=17 \
           -P first_filters=16 \
           -P input_size=14000 \
           -P lr_start=0.0101590069352232 \
           -P lr_power=5 \
           -P epochs=100 \
           -P csv_path_train=/beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets \
           -P csv_path_val=/beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN \
           -P scaler=l2 \
           -P n_levels=3 \
           -P pool_size=4
    
    2022/03/03 14:24:46 INFO mlflow.projects.utils: === Created directory /tmp/tmp7c_e9yu1 for downloading remote URIs passed to arguments of type 'path' ===
    2022/03/03 14:24:46 INFO mlflow.projects.backend.local: === Running command 'source /cluster/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-eaf130b8edd83d20d4f1e0db4286dabd625893fe 1>&2 && python src/fluotracify/train
    ing/train.py --fluotracify_path /beegfs/ye53nis/drmed-git/src --batch_size 17 --input_size 14000 --lr_start 0.0101590069352232 --lr_power 5 --epochs 100 --csv_path_train /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets --csv_p
    ath_val /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN --col_per_example 3 --scaler l2 --n_levels 3 --first_filters 16 --pool_size 4' in run with ID '34a6d207ac594035b1009c330fb67a65' ===
    2022-03-03 14:24:50.057413: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-03 14:24:54,004 - train -  Python version: 3.9.6 (default, Jul 30 2021, 16:35:19)
    [GCC 7.5.0]
    2022-03-03 14:24:54,004 - train -  Tensorflow version: 2.5.0
    2022-03-03 14:24:54,004 - train -  tf.keras version: 2.5.0
    2022-03-03 14:24:54,004 - train -  Cudnn version: 8
    2022-03-03 14:24:54,004 - train -  Cuda version: 11.2
    2022-03-03 14:24:54.006301: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
    2022-03-03 14:24:54.030317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-03-03 14:24:54.030405: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-03 14:24:54.038110: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-03-03 14:24:54.038189: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    2022-03-03 14:24:54.041129: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
    2022-03-03 14:24:54.042988: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
    2022-03-03 14:24:54.050220: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
    2022-03-03 14:24:54.052310: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
    2022-03-03 14:24:54.053334: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-03-03 14:24:54.056200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-03-03 14:24:54,056 - train -  GPUs: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]. Trying to set memory growth to "True"...
    2022-03-03 14:24:54,056 - train -  Setting memory growth successful.
    2022-03-03 14:25:01,220 - train -  1/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set004.csv
    2022-03-03 14:25:05,184 - train -  2/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set002.csv
    2022-03-03 14:25:08,951 - train -  3/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set001.csv
    2022-03-03 14:25:15,413 - train -  4/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.1/traces_brightclust_Nov2020_D1.0_set007.csv
    2022-03-03 14:25:33,960 - train -  5/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set002.csv
    2022-03-03 14:25:37,427 - train -  6/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/1.0/traces_brightclust_Nov2020_D10_set010.csv
    2022-03-03 14:25:41,118 - train -  7/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set004.csv
    2022-03-03 14:25:49,655 - train -  8/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.1/traces_brightclust_Nov2020_D3.0_set010.csv
    2022-03-03 14:25:54,052 - train -  9/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set009.csv
    2022-03-03 14:25:57,575 - train -  10/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set003.csv
    2022-03-03 14:26:01,255 - train -  11/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/1.0/traces_brightclust_Nov2020_D0.1_set007.csv
    2022-03-03 14:26:04,485 - train -  12/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/1.0/traces_brightclust_Nov2020_D1.0_set001.csv
    2022-03-03 14:26:08,073 - train -  13/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.01/traces_brightclust_Nov2020_D10_set003.csv
    2022-03-03 14:26:11,768 - train -  14/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/1.0/traces_brightclust_Nov2020_D0.6_set002.csv
    2022-03-03 14:26:19,400 - train -  15/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set005.csv
    2022-03-03 14:26:23,640 - train -  16/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/1.0/traces_brightclust_Nov2020_D0.069_set009.csv
    2022-03-03 14:26:27,567 - train -  17/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/1.0/traces_brightclust_Nov2020_D50_set004.csv
    2022-03-03 14:26:31,999 - train -  18/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set010.csv
    2022-03-03 14:26:36,569 - train -  19/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set009.csv
    2022-03-03 14:26:40,326 - train -  20/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/1.0/0.01/traces_brightclust_Nov2020_D1.0_set010.csv
    2022-03-03 14:26:43,920 - train -  21/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set007.csv
    2022-03-03 14:26:48,619 - train -  22/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/1.0/traces_brightclust_Nov2020_D0.2_set010.csv
    2022-03-03 14:26:53,742 - train -  23/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set001.csv
    2022-03-03 14:26:57,605 - train -  24/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set003.csv
    2022-03-03 14:27:01,008 - train -  25/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/1.0/traces_brightclust_Nov2020_D3.0_set003.csv
    2022-03-03 14:27:04,458 - train -  26/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set004.csv
    2022-03-03 14:27:07,860 - train -  27/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set001.csv
    2022-03-03 14:27:11,489 - train -  28/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set005.csv
    2022-03-03 14:27:15,309 - train -  29/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.1/traces_brightclust_Nov2020_D0.08_set006.csv
    2022-03-03 14:27:18,677 - train -  30/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.01/traces_brightclust_Nov2020_D0.4_set004.csv
    2022-03-03 14:27:22,854 - train -  31/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set006.csv
    2022-03-03 14:27:26,667 - train -  32/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.1/traces_brightclust_Nov2020_D0.2_set004.csv
    2022-03-03 14:27:30,369 - train -  33/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set005.csv
    2022-03-03 14:27:34,196 - train -  34/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set003.csv
    2022-03-03 14:27:37,882 - train -  35/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.2/0.01/traces_brightclust_Nov2020_D0.2_set003.csv
    2022-03-03 14:27:41,437 - train -  36/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.01/traces_brightclust_Nov2020_D50_set006.csv
    2022-03-03 14:27:45,344 - train -  37/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.1/traces_brightclust_Nov2020_D0.6_set004.csv
    2022-03-03 14:27:48,979 - train -  38/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/1.0/traces_brightclust_Nov2020_D0.08_set004.csv
    2022-03-03 14:27:52,611 - train -  39/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.6/0.01/traces_brightclust_Nov2020_D0.6_set010.csv
    2022-03-03 14:27:56,754 - train -  40/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/10/0.1/traces_brightclust_Nov2020_D10_set007.csv
    2022-03-03 14:28:00,273 - train -  41/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/1.0/traces_brightclust_Nov2020_D0.4_set006.csv
    2022-03-03 14:28:03,971 - train -  42/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.4/0.1/traces_brightclust_Nov2020_D0.4_set002.csv
    2022-03-03 14:28:07,564 - train -  43/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.1/0.01/traces_brightclust_Nov2020_D0.1_set006.csv
    2022-03-03 14:28:11,272 - train -  44/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/3.0/0.01/traces_brightclust_Nov2020_D3.0_set006.csv
    2022-03-03 14:28:15,163 - train -  45/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set010.csv
    2022-03-03 14:28:18,794 - train -  46/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.08/0.01/traces_brightclust_Nov2020_D0.08_set007.csv
    2022-03-03 14:28:22,428 - train -  47/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/0.069/0.1/traces_brightclust_Nov2020_D0.069_set002.csv
    2022-03-03 14:28:26,334 - train -  48/48: /beegfs/ye53nis/saves/firstartifact_Nov2020_train_max2sets/50/0.1/traces_brightclust_Nov2020_D50_set009.csv
    2022-03-03 14:28:30,056 - train -  1/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/0.1/traces_brightclust_Nov2020_D1.0_set009.csv
    2022-03-03 14:28:33,687 - train -  2/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/50/1.0/traces_brightclust_Nov2020_D50_set007.csv
    2022-03-03 14:28:37,128 - train -  3/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.4/0.1/traces_brightclust_Nov2020_D0.4_set009.csv
    2022-03-03 14:28:40,617 - train -  4/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/1.0/traces_brightclust_Nov2020_D3.0_set009.csv
    2022-03-03 14:28:44,105 - train -  5/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.1/0.01/traces_brightclust_Nov2020_D0.1_set008.csv
    2022-03-03 14:28:47,594 - train -  6/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/10/0.01/traces_brightclust_Nov2020_D10_set008.csv
    2022-03-03 14:28:51,273 - train -  7/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.08/0.1/traces_brightclust_Nov2020_D0.08_set008.csv
    2022-03-03 14:28:56,181 - train -  8/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/1.0/1.0/traces_brightclust_Nov2020_D1.0_set008.csv
    2022-03-03 14:28:59,593 - train -  9/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/3.0/0.01/traces_brightclust_Nov2020_D3.0_set008.csv
    2022-03-03 14:29:03,237 - train -  10/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.2/0.1/traces_brightclust_Nov2020_D0.2_set006.csv
    2022-03-03 14:29:07,571 - train -  11/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.069/0.1/traces_brightclust_Nov2020_D0.069_set006.csv
    2022-03-03 14:29:11,901 - train -  12/12: /beegfs/ye53nis/saves/firstartifact_Nov2020_val_max2sets_SORTEDIN/0.6/0.1/traces_brightclust_Nov2020_D0.6_set006.csv
    2022-03-03 14:29:12,117 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 4800), (16384, 4800), (16384, 4800)]
    2022-03-03 14:29:12,206 - train -  The given DataFrame was split into 3 parts with shapes: [(16384, 1200), (16384, 1200), (16384, 1200)]
    2022-03-03 14:29:12.413198: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operatio
    ns:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    2022-03-03 14:29:12.417391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
    pciBusID: 0000:82:00.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
    coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
    2022-03-03 14:29:12.419377: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
    2022-03-03 14:29:12.419477: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
    2022-03-03 14:29:12.833057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
    2022-03-03 14:29:12.833127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0
    2022-03-03 14:29:12.833141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N
    2022-03-03 14:29:12.835995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15403 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, p
    ci bus id: 0000:82:00.0, compute capability: 6.0)
    2022-03-03 14:29:14,768 - train -  number of examples: 4800
    2022-03-03 14:29:15,107 - train -  number of examples: 1200
    2022-03-03 14:29:16,570 - train -  unet: input shape: (None, None, 1), output shape: (None, None, 1)
    2022/03/03 14:29:16 WARNING mlflow.utils.autologging_utils: Encountered unexpected error during tensorflow autologging: Changing param values is not allowed. Param with key='batch_size' was already logged with value='17' for run ID='34a
    6d207ac594035b1009c330fb67a65'. Attempted logging new value 'None'.
    Epoch 1/100
    2022-03-03 14:29:23.930177: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
    2022-03-03 14:29:24.066046: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2199895000 Hz
    2022-03-03 14:29:33.256586: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
    2022-03-03 14:29:33.572296: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8101
    2022-03-03 14:29:34.075599: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
    2022-03-03 14:29:34.363317: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
    282/282 [==============================] - 45s 98ms/step - loss: 1.0922 - tp0.1: 8529810.0000 - fp0.1: 10393289.0000 - tn0.1: 57940184.0000 - fn0.1: 1681616.0000 - precision0.1: 0.4508 - recall0.1: 0.8353 - tp0.3: 7724105.0000 - fp0.3:
    3960382.0000 - tn0.3: 64373080.0000 - fn0.3: 2487321.0000 - precision0.3: 0.6611 - recall0.3: 0.7564 - tp0.5: 7024685.0000 - fp0.5: 1630985.0000 - tn0.5: 66702492.0000 - fn0.5: 3186741.0000 - precision0.5: 0.8116 - recall0.5: 0.6879 - t
    p0.7: 6159487.0000 - fp0.7: 585291.0000 - tn0.7: 67748208.0000 - fn0.7: 4051939.0000 - precision0.7: 0.9132 - recall0.7: 0.6032 - tp0.9: 4931800.0000 - fp0.9: 143363.0000 - tn0.9: 68190096.0000 - fn0.9: 5279626.0000 - precision0.9: 0.97
    18 - recall0.9: 0.4830 - accuracy: 0.9387 - auc: 0.9142 - f1: 0.7446 - val_loss: 3.2432 - val_tp0.1: 2699606.0000 - val_fp0.1: 16783988.0000 - val_tn0.1: 12739.0000 - val_fn0.1: 627.0000 - val_precision0.1: 0.1386 - val_recall0.1: 0.999
    8 - val_tp0.3: 2697002.0000 - val_fp0.3: 16748206.0000 - val_tn0.3: 48521.0000 - val_fn0.3: 3231.0000 - val_precision0.3: 0.1387 - val_recall0.3: 0.9988 - val_tp0.5: 2695536.0000 - val_fp0.5: 16737605.0000 - val_tn0.5: 59122.0000 - val_
    fn0.5: 4697.0000 - val_precision0.5: 0.1387 - val_recall0.5: 0.9983 - val_tp0.7: 2689329.0000 - val_fp0.7: 16678401.0000 - val_tn0.7: 118326.0000 - val_fn0.7: 10904.0000 - val_precision0.7: 0.1389 - val_recall0.7: 0.9960 - val_tp0.9: 22
    37862.0000 - val_fp0.9: 12480955.0000 - val_tn0.9: 4315772.0000 - val_fn0.9: 462371.0000 - val_precision0.9: 0.1520 - val_recall0.9: 0.8288 - val_accuracy: 0.1413 - val_auc: 0.5907 - val_f1: 0.2436
    
    ...
    
    Epoch 100/100
    282/282 [==============================] - 25s 88ms/step - loss: 0.2009 - tp0.1: 9736541.0000 - fp0.1: 3605117.0000 - tn0.1: 64740432.0000 - fn0.1: 462821.0000 - precision0.1: 0.7298 - recall0.1: 0.9546 - tp0.3: 9402084.0000 - fp0.3: 17
    39716.0000 - tn0.3: 66605812.0000 - fn0.3: 797278.0000 - precision0.3: 0.8439 - recall0.3: 0.9218 - tp0.5: 9016887.0000 - fp0.5: 920615.0000 - tn0.5: 67424920.0000 - fn0.5: 1182475.0000 - precision0.5: 0.9074 - recall0.5: 0.8841 - tp0.7
    : 8513773.0000 - fp0.7: 435436.0000 - tn0.7: 67910080.0000 - fn0.7: 1685589.0000 - precision0.7: 0.9513 - recall0.7: 0.8347 - tp0.9: 7585158.0000 - fp0.9: 106557.0000 - tn0.9: 68238992.0000 - fn0.9: 2614204.0000 - precision0.9: 0.9861 -
     recall0.9: 0.7437 - accuracy: 0.9732 - auc: 0.9746 - f1: 0.8956 - val_loss: 0.2670 - val_tp0.1: 2538570.0000 - val_fp0.1: 1039161.0000 - val_tn0.1: 15761772.0000 - val_fn0.1: 157457.0000 - val_precision0.1: 0.7095 - val_recall0.1: 0.94
    16 - val_tp0.3: 2435718.0000 - val_fp0.3: 590706.0000 - val_tn0.3: 16210227.0000 - val_fn0.3: 260309.0000 - val_precision0.3: 0.8048 - val_recall0.3: 0.9034 - val_tp0.5: 2326279.0000 - val_fp0.5: 379342.0000 - val_tn0.5: 16421591.0000 -
     val_fn0.5: 369748.0000 - val_precision0.5: 0.8598 - val_recall0.5: 0.8629 - val_tp0.7: 2189825.0000 - val_fp0.7: 228353.0000 - val_tn0.7: 16572580.0000 - val_fn0.7: 506202.0000 - val_precision0.7: 0.9056 - val_recall0.7: 0.8122 - val_t
    p0.9: 1947380.0000 - val_fp0.9: 92879.0000 - val_tn0.9: 16708054.0000 - val_fn0.9: 748647.0000 - val_precision0.9: 0.9545 - val_recall0.9: 0.7223 - val_accuracy: 0.9616 - val_auc: 0.9652 - val_f1: 0.8613
    2022-03-03 15:16:02.951715: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
    2022/03/03 15:16:13 INFO mlflow.projects: === Run (ID '34a6d207ac594035b1009c330fb67a65') succeeded ===
    
    
    • name of run: 34a6d207ac594035b1009c330fb67a65
    • metrics after 100th epoch:
      • precisionval,0.5: 0.8598 - recallval,0.5: 0.8629 - f1val,0.5: 0.8613
      • aucval: 0.9652
  10. all metrics after 100th epoch with hparams
    run aucval f1val,0.5 precval,0.5 recallval,0.5 model size hpbatchsize hpfirstfilters hpinputsize hplrpower hplrstart hpnlevels hppoolsize hpscaler
    484af471c61943fa90e5f78e78a229f0 0.9814 0.9187 0.9091 0.9285 275 MB 26 44 16384 (here: 14000) 1 0.0136170138242663 7 2 standard
    0cd2023eeaf745aca0d3e8ad5e1fc653 0.9818 0.9069 0.8955 0.9185 200 MB 15 23 16384 (here: 14000) 7 0.0305060808685107 6 4 quantg
    fe81d71c52404ed790b3a32051258da9 0.9849 0.9260 0.9184 0.9338 186 MB 20 78 16384 (here: 14000) 4 0.0584071108418767 4 4 standard
    ff67be0b68e540a9a29a36a2d0c7a5be 0.9859 0.9298 0.9230 0.9367 14 MB 28 6 16384 (here: 14000) 1 0.0553313915596308 5 4 minmax
    19e3e786e1bc4e2b93856f5dc9de8216 0.9595 0.8911 0.8983 0.8839 172 MB 20 128 16384 (here: 14000) 1 0.043549707353273 3 4 standard
    347669d050f344ad9fb9e480c814f727 0.9848 0.9246 0.9254 0.9238 73 MB 10 16 8192 (here: 14000) 1 0.0627676336651573 5 4 robust
    c1204e3a8a1e4c40a35b5b7b1922d1ce 0.9858 0.9207 0.9179 0.9234 312 MB 14 16 16384 (here: 14000) 5 0.0192390310290551 9 2 robust
    714af8cd12c1441eac4ca980e8c20070 0.9843 0.9304 0.9257 0.9352 234 MB 9 64 4096 (here: 14000) 1 0.0100697459464075 5 4 maxabs
    34a6d207ac594035b1009c330fb67a65 0.9652 0.8613 0.8598 0.8629 7 MB 17 16 16384 (here: 14000) 5 0.0101590069352232 3 4 l2

    :END:

2.6.9 Analysis 1: show model architecture and loss plots

  • first load modules and define model lists and dictionaries to connect hparam runs and final runs.
    %cd ~/Programme/drmed-git
    
    /home/lex/Programme/drmed-git
    
    import datetime
    import logging
    import multipletau
    import os
    import sys
    
    import matplotlib.pyplot as plt
    import numpy as np
    import pandas as pd
    import seaborn as sns
    import tensorflow as tf
    
    from pathlib import Path
    from pprint import pprint
    from tensorflow.keras.optimizers import Adam
    from mlflow.keras import load_model
    
    FLUOTRACIFY_PATH = '/home/lex/Programme/drmed-git/src/'
    sys.path.append(FLUOTRACIFY_PATH)
    from fluotracify.training import (build_model as bm)
    
    model_ls = ['ff67be0b68e540a9a29a36a2d0c7a5be', '347669d050f344ad9fb9e480c814f727',
                '714af8cd12c1441eac4ca980e8c20070', '34a6d207ac594035b1009c330fb67a65',
                '484af471c61943fa90e5f78e78a229f0', '0cd2023eeaf745aca0d3e8ad5e1fc653',
                'fe81d71c52404ed790b3a32051258da9', '19e3e786e1bc4e2b93856f5dc9de8216',
                'c1204e3a8a1e4c40a35b5b7b1922d1ce']
    model_dict = {
        ('9051e32b87d84f3485b980067addec30', '61ff87bdb89b4e2ba64f8dacc774992d') :  '484af471c61943fa90e5f78e78a229f0',
        ('93b168c0ff7942c8a908a94129daf973', 'f243b3b742de4dbcb7ccfbd4244706f8') :  '0cd2023eeaf745aca0d3e8ad5e1fc653',
        ('a5b8551144ff46e697a39cd1551e1475', '98cf8cdef9c54b5286e277e75e2ab8c1') :  'fe81d71c52404ed790b3a32051258da9',
        ('00f2635d9fa2463c9a066722163405be', 'd0a8e1748b194f3290d471b6b44f19f8') :  'ff67be0b68e540a9a29a36a2d0c7a5be',
        ('5604d43c1ece461b8e6eaa0dfb65d6dc', '3612536a77f34f22bc83d1d809140aa6') :  '19e3e786e1bc4e2b93856f5dc9de8216',
        ('7cafab027cdd4fc9bf20a43e989df510', '16dff15d935f45e2a836b1f41b07b4e3') :  '347669d050f344ad9fb9e480c814f727',
        ('0e328920e86049928202db95e8cfb7be', 'bf9d2725eb16462d9a101f0a077ce2b5') :  'c1204e3a8a1e4c40a35b5b7b1922d1ce',
        ('1c954fbc02b747bc813c587ac703c74a', 'ba49a80c2616407a8f1fe1fd12096fe0') :  '714af8cd12c1441eac4ca980e8c20070',
        ('3cbd945b62ec4634839372e403f6f377', '458b36a70db843719d202a8eda448f17') :  '34a6d207ac594035b1009c330fb67a65'}
    
    model_name_ls = [f'{s:.5}' for s in model_ls]
    
    pred_thresh = 0.5
    
    
  • now print model details for each model where it was saved
    for i in model_ls:
        print(f'model: {i}')
        try:
            logged_scaler = Path(f'/home/lex/Programme/drmed-git/data/mlruns/10/{i}/params/scaler')
            logged_scaler = !cat $logged_scaler
            logged_scaler = logged_scaler[0]
    
            logged_model = Path(f'/home/lex/Programme/drmed-git/data/mlruns/10/{i}/artifacts/model')
            logged_model = load_model(logged_model, compile=False)
            logged_model.compile(loss=bm.binary_ce_dice_loss(),
                                 optimizer=Adam(),
                                 metrics = bm.unet_metrics([0.1, 0.3, 0.5, 0.7, 0.9]))
            print(logged_model.summary())
    
        except AttributeError:
            pass
        print('----------------------------------------')
    
    model: ff67be0b68e540a9a29a36a2d0c7a5be
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    2022-08-23 14:27:17,992 - build model -  Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    Model: "unet_depth5"
    __________________________________________________________________________________________________
     Layer (type)                   Output Shape         Param #     Connected to
    ==================================================================================================
     input_1 (InputLayer)           [(None, None, 1)]    0           []
    
     encode0 (Sequential)           (None, None, 6)      186         ['input_1[0][0]']
    
     mp_encode0 (MaxPooling1D)      (None, None, 6)      0           ['encode0[0][0]']
    
     encode1 (Sequential)           (None, None, 12)     768         ['mp_encode0[0][0]']
    
     mp_encode1 (MaxPooling1D)      (None, None, 12)     0           ['encode1[0][0]']
    
     encode2 (Sequential)           (None, None, 24)     2832        ['mp_encode1[0][0]']
    
     mp_encode2 (MaxPooling1D)      (None, None, 24)     0           ['encode2[0][0]']
    
     encode3 (Sequential)           (None, None, 48)     10848       ['mp_encode2[0][0]']
    
     mp_encode3 (MaxPooling1D)      (None, None, 48)     0           ['encode3[0][0]']
    
     encode4 (Sequential)           (None, None, 96)     42432       ['mp_encode3[0][0]']
    
     mp_encode4 (MaxPooling1D)      (None, None, 96)     0           ['encode4[0][0]']
    
     two_conv_center (Sequential)   (None, None, 192)    167808      ['mp_encode4[0][0]']
    
     conv_transpose_decoder4 (Seque  (None, None, 192)   148416      ['two_conv_center[0][0]']
     ntial)
    
     decoder4 (Concatenate)         (None, None, 288)    0           ['encode4[0][0]',
                                                                      'conv_transpose_decoder4[0][0]']
    
     two_conv_decoder4 (Sequential)  (None, None, 192)   278400      ['decoder4[0][0]']
    
     conv_transpose_decoder3 (Seque  (None, None, 96)    74208       ['two_conv_decoder4[0][0]']
     ntial)
    
     decoder3 (Concatenate)         (None, None, 144)    0           ['encode3[0][0]',
                                                                      'conv_transpose_decoder3[0][0]']
    
     two_conv_decoder3 (Sequential)  (None, None, 96)    70080       ['decoder3[0][0]']
    
     conv_transpose_decoder2 (Seque  (None, None, 48)    18672       ['two_conv_decoder3[0][0]']
     ntial)
    
     decoder2 (Concatenate)         (None, None, 72)     0           ['encode2[0][0]',
                                                                      'conv_transpose_decoder2[0][0]']
    
     two_conv_decoder2 (Sequential)  (None, None, 48)    17760       ['decoder2[0][0]']
    
     conv_transpose_decoder1 (Seque  (None, None, 24)    4728        ['two_conv_decoder2[0][0]']
     ntial)
    
     decoder1 (Concatenate)         (None, None, 36)     0           ['encode1[0][0]',
                                                                      'conv_transpose_decoder1[0][0]']
    
     two_conv_decoder1 (Sequential)  (None, None, 24)    4560        ['decoder1[0][0]']
    
     conv_transpose_decoder0 (Seque  (None, None, 12)    1212        ['two_conv_decoder1[0][0]']
     ntial)
    
     decoder0 (Concatenate)         (None, None, 18)     0           ['encode0[0][0]',
                                                                      'conv_transpose_decoder0[0][0]']
    
     two_conv_decoder0 (Sequential)  (None, None, 12)    1200        ['decoder0[0][0]']
    
     conv1d_22 (Conv1D)             (None, None, 1)      13          ['two_conv_decoder0[0][0]']
    
    ==================================================================================================
    Total params: 844,123
    Trainable params: 840,379
    Non-trainable params: 3,744
    __________________________________________________________________________________________________
    None
    ----------------------------------------
    model: 347669d050f344ad9fb9e480c814f727WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    2022-08-23 14:27:26,761 - build model -  Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    Model: "unet_depth5"
    __________________________________________________________________________________________________
     Layer (type)                   Output Shape         Param #     Connected to
    ==================================================================================================
     input_1 (InputLayer)           [(None, None, 1)]    0           []
    
     encode0 (Sequential)           (None, None, 16)     976         ['input_1[0][0]']
    
     mp_encode0 (MaxPooling1D)      (None, None, 16)     0           ['encode0[0][0]']
    
     encode1 (Sequential)           (None, None, 32)     4928        ['mp_encode0[0][0]']
    
     mp_encode1 (MaxPooling1D)      (None, None, 32)     0           ['encode1[0][0]']
    
     encode2 (Sequential)           (None, None, 64)     19072       ['mp_encode1[0][0]']
    
     mp_encode2 (MaxPooling1D)      (None, None, 64)     0           ['encode2[0][0]']
    
     encode3 (Sequential)           (None, None, 128)    75008       ['mp_encode2[0][0]']
    
     mp_encode3 (MaxPooling1D)      (None, None, 128)    0           ['encode3[0][0]']
    
     encode4 (Sequential)           (None, None, 256)    297472      ['mp_encode3[0][0]']
    
     mp_encode4 (MaxPooling1D)      (None, None, 256)    0           ['encode4[0][0]']
    
     two_conv_center (Sequential)   (None, None, 512)    1184768     ['mp_encode4[0][0]']
    
     conv_transpose_decoder4 (Seque  (None, None, 512)   1051136     ['two_conv_center[0][0]']
     ntial)
    
     decoder4 (Concatenate)         (None, None, 768)    0           ['encode4[0][0]',
                                                                      'conv_transpose_decoder4[0][0]']
    
     two_conv_decoder4 (Sequential)  (None, None, 512)   1971200     ['decoder4[0][0]']
    
     conv_transpose_decoder3 (Seque  (None, None, 256)   525568      ['two_conv_decoder4[0][0]']
     ntial)
    
     decoder3 (Concatenate)         (None, None, 384)    0           ['encode3[0][0]',
                                                                      'conv_transpose_decoder3[0][0]']
    
     two_conv_decoder3 (Sequential)  (None, None, 256)   494080      ['decoder3[0][0]']
    
     conv_transpose_decoder2 (Seque  (None, None, 128)   131712      ['two_conv_decoder3[0][0]']
     ntial)
    
     decoder2 (Concatenate)         (None, None, 192)    0           ['encode2[0][0]',
                                                                      'conv_transpose_decoder2[0][0]']
    
     two_conv_decoder2 (Sequential)  (None, None, 128)   124160      ['decoder2[0][0]']
    
     conv_transpose_decoder1 (Seque  (None, None, 64)    33088       ['two_conv_decoder2[0][0]']
     ntial)
    
     decoder1 (Concatenate)         (None, None, 96)     0           ['encode1[0][0]',
                                                                      'conv_transpose_decoder1[0][0]']
    
     two_conv_decoder1 (Sequential)  (None, None, 64)    31360       ['decoder1[0][0]']
    
     conv_transpose_decoder0 (Seque  (None, None, 32)    8352        ['two_conv_decoder1[0][0]']
     ntial)
    
     decoder0 (Concatenate)         (None, None, 48)     0           ['encode0[0][0]',
                                                                      'conv_transpose_decoder0[0][0]']
    
     two_conv_decoder0 (Sequential)  (None, None, 32)    8000        ['decoder0[0][0]']
    
     conv1d_22 (Conv1D)             (None, None, 1)      33          ['two_conv_decoder0[0][0]']
    
    ==================================================================================================
    Total params: 5,960,913
    Trainable params: 5,950,929
    Non-trainable params: 9,984
    __________________________________________________________________________________________________
    None
    ----------------------------------------
    model: 714af8cd12c1441eac4ca980e8c20070
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    2022-08-23 14:27:35,085 - build model -  Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    Model: "unet_depth5"
    __________________________________________________________________________________________________
     Layer (type)                   Output Shape         Param #     Connected to
    ==================================================================================================
     input_1 (InputLayer)           [(None, None, 1)]    0           []
    
     encode0 (Sequential)           (None, None, 64)     13120       ['input_1[0][0]']
    
     mp_encode0 (MaxPooling1D)      (None, None, 64)     0           ['encode0[0][0]']
    
     encode1 (Sequential)           (None, None, 128)    75008       ['mp_encode0[0][0]']
    
     mp_encode1 (MaxPooling1D)      (None, None, 128)    0           ['encode1[0][0]']
    
     encode2 (Sequential)           (None, None, 256)    297472      ['mp_encode1[0][0]']
    
     mp_encode2 (MaxPooling1D)      (None, None, 256)    0           ['encode2[0][0]']
    
     encode3 (Sequential)           (None, None, 512)    1184768     ['mp_encode2[0][0]']
    
     mp_encode3 (MaxPooling1D)      (None, None, 512)    0           ['encode3[0][0]']
    
     encode4 (Sequential)           (None, None, 512)    1577984     ['mp_encode3[0][0]']
    
     mp_encode4 (MaxPooling1D)      (None, None, 512)    0           ['encode4[0][0]']
    
     two_conv_center (Sequential)   (None, None, 1024)   4728832     ['mp_encode4[0][0]']
    
     conv_transpose_decoder4 (Seque  (None, None, 512)   2099712     ['two_conv_center[0][0]']
     ntial)
    
     decoder4 (Concatenate)         (None, None, 1024)   0           ['encode4[0][0]',
                                                                      'conv_transpose_decoder4[0][0]']
    
     two_conv_decoder4 (Sequential)  (None, None, 512)   2364416     ['decoder4[0][0]']
    
     conv_transpose_decoder3 (Seque  (None, None, 512)   1051136     ['two_conv_decoder4[0][0]']
     ntial)
    
     decoder3 (Concatenate)         (None, None, 1024)   0           ['encode3[0][0]',
                                                                      'conv_transpose_decoder3[0][0]']
    
     two_conv_decoder3 (Sequential)  (None, None, 512)   2364416     ['decoder3[0][0]']
    
     conv_transpose_decoder2 (Seque  (None, None, 512)   1051136     ['two_conv_decoder3[0][0]']
     ntial)
    
     decoder2 (Concatenate)         (None, None, 768)    0           ['encode2[0][0]',
                                                                      'conv_transpose_decoder2[0][0]']
    
     two_conv_decoder2 (Sequential)  (None, None, 512)   1971200     ['decoder2[0][0]']
    
     conv_transpose_decoder1 (Seque  (None, None, 256)   525568      ['two_conv_decoder2[0][0]']
     ntial)
    
     decoder1 (Concatenate)         (None, None, 384)    0           ['encode1[0][0]',
                                                                      'conv_transpose_decoder1[0][0]']
    
     two_conv_decoder1 (Sequential)  (None, None, 256)   494080      ['decoder1[0][0]']
    
     conv_transpose_decoder0 (Seque  (None, None, 128)   131712      ['two_conv_decoder1[0][0]']
     ntial)
    
     decoder0 (Concatenate)         (None, None, 192)    0           ['encode0[0][0]',
                                                                      'conv_transpose_decoder0[0][0]']
    
     two_conv_decoder0 (Sequential)  (None, None, 128)   124160      ['decoder0[0][0]']
    
     conv1d_22 (Conv1D)             (None, None, 1)      129         ['two_conv_decoder0[0][0]']
    
    ==================================================================================================
    Total params: 20,054,849
    Trainable params: 20,033,345
    Non-trainable params: 21,504
    __________________________________________________________________________________________________
    None
    ----------------------------------------
    model: 34a6d207ac594035b1009c330fb67a65
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    2022-08-23 14:27:43,275 - build model -  Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    Model: "unet_depth3"
    __________________________________________________________________________________________________
     Layer (type)                   Output Shape         Param #     Connected to
    ==================================================================================================
     input_1 (InputLayer)           [(None, None, 1)]    0           []
    
     encode0 (Sequential)           (None, None, 16)     976         ['input_1[0][0]']
    
     mp_encode0 (MaxPooling1D)      (None, None, 16)     0           ['encode0[0][0]']
    
     encode1 (Sequential)           (None, None, 32)     4928        ['mp_encode0[0][0]']
    
     mp_encode1 (MaxPooling1D)      (None, None, 32)     0           ['encode1[0][0]']
    
     encode2 (Sequential)           (None, None, 64)     19072       ['mp_encode1[0][0]']
    
     mp_encode2 (MaxPooling1D)      (None, None, 64)     0           ['encode2[0][0]']
    
     two_conv_center (Sequential)   (None, None, 128)    75008       ['mp_encode2[0][0]']
    
     conv_transpose_decoder2 (Seque  (None, None, 128)   66176       ['two_conv_center[0][0]']
     ntial)
    
     decoder2 (Concatenate)         (None, None, 192)    0           ['encode2[0][0]',
                                                                      'conv_transpose_decoder2[0][0]']
    
     two_conv_decoder2 (Sequential)  (None, None, 128)   124160      ['decoder2[0][0]']
    
     conv_transpose_decoder1 (Seque  (None, None, 64)    33088       ['two_conv_decoder2[0][0]']
     ntial)
    
     decoder1 (Concatenate)         (None, None, 96)     0           ['encode1[0][0]',
                                                                      'conv_transpose_decoder1[0][0]']
    
     two_conv_decoder1 (Sequential)  (None, None, 64)    31360       ['decoder1[0][0]']
    
     conv_transpose_decoder0 (Seque  (None, None, 32)    8352        ['two_conv_decoder1[0][0]']
     ntial)
    
     decoder0 (Concatenate)         (None, None, 48)     0           ['encode0[0][0]',
                                                                      'conv_transpose_decoder0[0][0]']
    
     two_conv_decoder0 (Sequential)  (None, None, 32)    8000        ['decoder0[0][0]']
    
     conv1d_14 (Conv1D)             (None, None, 1)      33          ['two_conv_decoder0[0][0]']
    
    ==================================================================================================
    Total params: 371,153
    Trainable params: 368,849
    Non-trainable params: 2,304
    __________________________________________________________________________________________________
    None
    ----------------------------------------
    model: 484af471c61943fa90e5f78e78a229f0
    ----------------------------------------
    model: 0cd2023eeaf745aca0d3e8ad5e1fc653
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    2022-08-23 14:27:49,590 - build model -  Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    Model: "unet_depth6"
    __________________________________________________________________________________________________
     Layer (type)                   Output Shape         Param #     Connected to
    ==================================================================================================
     input_1 (InputLayer)           [(None, None, 1)]    0           []
    
     encode0 (Sequential)           (None, None, 23)     1886        ['input_1[0][0]']
    
     mp_encode0 (MaxPooling1D)      (None, None, 23)     0           ['encode0[0][0]']
    
     encode1 (Sequential)           (None, None, 46)     9982        ['mp_encode0[0][0]']
    
     mp_encode1 (MaxPooling1D)      (None, None, 46)     0           ['encode1[0][0]']
    
     encode2 (Sequential)           (None, None, 92)     39008       ['mp_encode1[0][0]']
    
     mp_encode2 (MaxPooling1D)      (None, None, 92)     0           ['encode2[0][0]']
    
     encode3 (Sequential)           (None, None, 184)    154192      ['mp_encode2[0][0]']
    
     mp_encode3 (MaxPooling1D)      (None, None, 184)    0           ['encode3[0][0]']
    
     encode4 (Sequential)           (None, None, 368)    613088      ['mp_encode3[0][0]']
    
     mp_encode4 (MaxPooling1D)      (None, None, 368)    0           ['encode4[0][0]']
    
     encode5 (Sequential)           (None, None, 512)    1356800     ['mp_encode4[0][0]']
    
     mp_encode5 (MaxPooling1D)      (None, None, 512)    0           ['encode5[0][0]']
    
     two_conv_center (Sequential)   (None, None, 1024)   4728832     ['mp_encode5[0][0]']
    
     conv_transpose_decoder5 (Seque  (None, None, 512)   2099712     ['two_conv_center[0][0]']
     ntial)
    
     decoder5 (Concatenate)         (None, None, 1024)   0           ['encode5[0][0]',
                                                                      'conv_transpose_decoder5[0][0]']
    
     two_conv_decoder5 (Sequential)  (None, None, 512)   2364416     ['decoder5[0][0]']
    
     conv_transpose_decoder4 (Seque  (None, None, 512)   1051136     ['two_conv_decoder5[0][0]']
     ntial)
    
     decoder4 (Concatenate)         (None, None, 880)    0           ['encode4[0][0]',
                                                                      'conv_transpose_decoder4[0][0]']
    
     two_conv_decoder4 (Sequential)  (None, None, 512)   2143232     ['decoder4[0][0]']
    
     conv_transpose_decoder3 (Seque  (None, None, 368)   755504      ['two_conv_decoder4[0][0]']
     ntial)
    
     decoder3 (Concatenate)         (None, None, 552)    0           ['encode3[0][0]',
                                                                      'conv_transpose_decoder3[0][0]']
    
     two_conv_decoder3 (Sequential)  (None, None, 368)   1019360     ['decoder3[0][0]']
    
     conv_transpose_decoder2 (Seque  (None, None, 184)   271768      ['two_conv_decoder3[0][0]']
     ntial)
    
     decoder2 (Concatenate)         (None, None, 276)    0           ['encode2[0][0]',
                                                                      'conv_transpose_decoder2[0][0]']
    
     two_conv_decoder2 (Sequential)  (None, None, 184)   255760      ['decoder2[0][0]']
    
     conv_transpose_decoder1 (Seque  (None, None, 92)    68172       ['two_conv_decoder2[0][0]']
     ntial)
    
     decoder1 (Concatenate)         (None, None, 138)    0           ['encode1[0][0]',
                                                                      'conv_transpose_decoder1[0][0]']
    
     two_conv_decoder1 (Sequential)  (None, None, 92)    64400       ['decoder1[0][0]']
    
     conv_transpose_decoder0 (Seque  (None, None, 46)    17158       ['two_conv_decoder1[0][0]']
     ntial)
    
     decoder0 (Concatenate)         (None, None, 69)     0           ['encode0[0][0]',
                                                                      'conv_transpose_decoder0[0][0]']
    
     two_conv_decoder0 (Sequential)  (None, None, 46)    16330       ['decoder0[0][0]']
    
     conv1d_26 (Conv1D)             (None, None, 1)      47          ['two_conv_decoder0[0][0]']
    
    ==================================================================================================
    Total params: 17,030,783
    Trainable params: 17,011,503
    Non-trainable params: 19,280
    __________________________________________________________________________________________________
    None
    ----------------------------------------
    model: fe81d71c52404ed790b3a32051258da9
    ----------------------------------------
    model: 19e3e786e1bc4e2b93856f5dc9de8216
    ----------------------------------------
    model: c1204e3a8a1e4c40a35b5b7b1922d1ce
    ----------------------------------------
    
  • plot loss vs validation loss for models in hyperparameter training. Unfortunately, this info was not saved out for the final runs due to a misconfiguration in mlflow. So for the final runs print the final train loss and validation loss
    hparam = run1_2_my.loc[cond2 & cond3]
    hparam.rename(index=model_dict)
    
    fig, axs = plt.subplots(len(hparam), 2, facecolor='white', figsize=(9, len(hparam)*3),
                            sharex=True, tight_layout=True)
    for i, idx in enumerate(hparam.index):
        for j in range(2):
            print(i, idx[j], model_dict[idx])
            try:
                logged_val_loss = Path(f'/beegfs/ye53nis/drmed-git/data/mlruns/8/{idx[j]}/metrics/val_loss')
                logged_val_loss = !cat $logged_val_loss
                logged_val_loss = [i.split(' ') for i in logged_val_loss]
                logged_val_loss = pd.DataFrame(logged_val_loss, columns=['time', 'val_loss', 'epoch']).astype(float)
    
                logged_train_loss = Path(f'/beegfs/ye53nis/drmed-git/data/mlruns/8/{idx[j]}/metrics/loss')
                logged_train_loss = !cat $logged_train_loss
                logged_train_loss = [i.split(' ') for i in logged_train_loss]
                logged_train_loss = pd.DataFrame(logged_train_loss, columns=['time', 'train_loss', 'epoch']).astype(float)
    
                losses = pd.DataFrame([logged_train_loss.iloc[-1, 1], logged_val_loss.iloc[-1, 1]], index=['train loss', 'val loss'],
                                      columns=[f'{model_dict[idx]}_{j}'])
                display(losses)
                last_losses = pd.concat([last_losses, ])
                sns.lineplot('epoch', 'train_loss', data=logged_train_loss, ax=axs[i, j], label='train loss')
                sns.lineplot('epoch', 'val_loss', data=logged_val_loss, ax=axs[i, j], label='val loss')
                if j == 0:
                    axs[i, j].set_title(model_dict[idx])
            except ValueError:
                pass
    plt.setp(axs, yscale='log', ylabel='')
    fig.align_ylabels(axs)
    
    0 9051e32b87d84f3485b980067addec30 484af471c61943fa90e5f78e78a229f0
    
      484af471c61943fa90e5f78e78a229f00
    train loss 0.196100
    val loss 0.199457
    0 61ff87bdb89b4e2ba64f8dacc774992d 484af471c61943fa90e5f78e78a229f0
    
      484af471c61943fa90e5f78e78a229f01
    train loss 0.194301
    val loss 0.197833
    1 93b168c0ff7942c8a908a94129daf973 0cd2023eeaf745aca0d3e8ad5e1fc653
    
      0cd2023eeaf745aca0d3e8ad5e1fc6530
    train loss 0.245977
    val loss 0.250499
    1 f243b3b742de4dbcb7ccfbd4244706f8 0cd2023eeaf745aca0d3e8ad5e1fc653
    
      0cd2023eeaf745aca0d3e8ad5e1fc6531
    train loss 0.278278
    val loss 0.283107
    2 a5b8551144ff46e697a39cd1551e1475 fe81d71c52404ed790b3a32051258da9
    
      fe81d71c52404ed790b3a32051258da90
    train loss 0.185773
    val loss 0.186789
    2 98cf8cdef9c54b5286e277e75e2ab8c1 fe81d71c52404ed790b3a32051258da9
    
      fe81d71c52404ed790b3a32051258da91
    train loss 0.183874
    val loss 0.184316
    3 00f2635d9fa2463c9a066722163405be ff67be0b68e540a9a29a36a2d0c7a5be
    
      ff67be0b68e540a9a29a36a2d0c7a5be0
    train loss 0.172448
    val loss 0.178439
    3 d0a8e1748b194f3290d471b6b44f19f8 ff67be0b68e540a9a29a36a2d0c7a5be
    4 5604d43c1ece461b8e6eaa0dfb65d6dc 19e3e786e1bc4e2b93856f5dc9de8216
    
      19e3e786e1bc4e2b93856f5dc9de82160
    train loss 0.250983
    val loss 0.260167
    4 3612536a77f34f22bc83d1d809140aa6 19e3e786e1bc4e2b93856f5dc9de8216
    
      19e3e786e1bc4e2b93856f5dc9de82161
    train loss 0.246347
    val loss 0.242849
    5 7cafab027cdd4fc9bf20a43e989df510 347669d050f344ad9fb9e480c814f727
    
      347669d050f344ad9fb9e480c814f7270
    train loss 0.194196
    val loss 0.221606
    5 16dff15d935f45e2a836b1f41b07b4e3 347669d050f344ad9fb9e480c814f727
    
      347669d050f344ad9fb9e480c814f7271
    train loss 0.196745
    val loss 0.250257
    6 0e328920e86049928202db95e8cfb7be c1204e3a8a1e4c40a35b5b7b1922d1ce
    
      c1204e3a8a1e4c40a35b5b7b1922d1ce0
    train loss 0.266620
    val loss 0.279587
    6 bf9d2725eb16462d9a101f0a077ce2b5 c1204e3a8a1e4c40a35b5b7b1922d1ce
    
      c1204e3a8a1e4c40a35b5b7b1922d1ce1
    train loss 0.262320
    val loss 0.265113
    7 1c954fbc02b747bc813c587ac703c74a 714af8cd12c1441eac4ca980e8c20070
    
      714af8cd12c1441eac4ca980e8c200700
    train loss 0.310728
    val loss 0.317540
    7 ba49a80c2616407a8f1fe1fd12096fe0 714af8cd12c1441eac4ca980e8c20070
    
      714af8cd12c1441eac4ca980e8c200701
    train loss 0.303487
    val loss 0.340323
    8 3cbd945b62ec4634839372e403f6f377 34a6d207ac594035b1009c330fb67a65
    
      34a6d207ac594035b1009c330fb67a650
    train loss 0.189994
    val loss 0.249494
    8 458b36a70db843719d202a8eda448f17 34a6d207ac594035b1009c330fb67a65
    
      34a6d207ac594035b1009c330fb67a651
    train loss 0.211194
    val loss 0.239334

    analysis1_train-loss-vs-val-loss.png

2.6.10 Experiment 2: Evaluate models on simulated test data

  • go to correct folder, load modules
    %cd /beegfs/ye53nis/drmed-git
    
    /beegfs/ye53nis/drmed-git
    
    import datetime
    import logging
    import multipletau
    import os
    import sys
    
    import matplotlib.pyplot as plt
    import numpy as np
    import pandas as pd
    import seaborn as sns
    import tensorflow as tf
    
    from pathlib import Path
    from pprint import pprint
    from tensorflow.keras.optimizers import Adam
    from mlflow.keras import load_model
    
    FLUOTRACIFY_PATH = '/beegfs/ye53nis/drmed-git/src/'
    sys.path.append(FLUOTRACIFY_PATH)
    from fluotracify.applications import (corr_fit_object as cfo,
                                          correction,
                                          correlate)
    from fluotracify.imports import ptu_utils as ptu
    from fluotracify.training import (build_model as bm,
                                      preprocess_data as ppd)
    from fluotracify.simulations import (
       import_simulation_from_csv as isfc,
       analyze_simulations as ans,
    )
    
    logging.basicConfig(filename="/beegfs/ye53nis/drmed-git/data/exp-220227-unet/jupyter.log",
                        format='%(asctime)s - %(message)s',
                        filemode='w',
                        force=True)
    log = logging.getLogger(__name__)
    log.setLevel(logging.DEBUG)
    
    sns.set_theme(style="whitegrid", font_scale=2, palette='colorblind',
                  context='paper')
    
    model_ls = ['ff67be0b68e540a9a29a36a2d0c7a5be', '347669d050f344ad9fb9e480c814f727',
                '714af8cd12c1441eac4ca980e8c20070', '34a6d207ac594035b1009c330fb67a65',
                '484af471c61943fa90e5f78e78a229f0', '0cd2023eeaf745aca0d3e8ad5e1fc653',
                'fe81d71c52404ed790b3a32051258da9', '19e3e786e1bc4e2b93856f5dc9de8216',
                'c1204e3a8a1e4c40a35b5b7b1922d1ce']
    
    model_name_ls = [f'{s:.5}' for s in model_ls]
    
    pred_thresh = 0.5
    
    2022-08-24 10:31:45.084478: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2022-08-24 10:31:45.084513: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    
    import importlib
    importlib.reload(ppd)
    importlib.reload(cfo)
    
    <module 'fluotracify.applications.corr_fit_object' from '/beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py'>
    
  • Load test data from simulation experiments
    col_per_example = 3
    lab_thresh = 0.04
    artifact = 0
    model_type = 1
    fwhm = 250
    
    output_path = "/beegfs/ye53nis/drmed-git/data/exp-220227-unet/2022-04-22_simulations/"
    sim_path = Path('/beegfs/ye53nis/saves/firstartifact_Nov2020_test')
    
    sim, _, nsamples, sim_params = isfc.import_from_csv(
        folder=sim_path,
        header=12,
        frac_train=1,
        col_per_example=col_per_example,
        dropindex=None,
        dropcolumns=None)
    
    diffrates = sim_params.loc['diffusion rate of molecules in micrometer^2 / s'].astype(np.float32)
    nmols = sim_params.loc['number of fast molecules'].astype(np.float32)
    clusters = sim_params.loc['diffusion rate of clusters in micrometer^2 / s'].astype(np.float32)
    sim_columns = [f'{d:.4}-{c:.4}' for d, c in zip(
        np.repeat(diffrates, nsamples[0]),
        np.repeat(clusters, nsamples[0]))]
    
    sim_sep = isfc.separate_data_and_labels(array=sim,
                                            nsamples=nsamples,
                                            col_per_example=col_per_example)
    sim_dirty = sim_sep['0']
    sim_dirty.columns = sim_columns
    
    sim_labels = sim_sep['1']
    sim_labels.columns = sim_columns
    sim_labbool = sim_labels > lab_thresh
    sim_labbool.columns = sim_columns
    sim_clean = sim_sep['2']
    sim_clean.columns = sim_columns
    
    sim_dirty
    
      0.069-1.0 0.069-1.0 0.069-1.0 0.069-1.0 0.069-1.0 0.069-1.0 0.069-1.0 0.069-1.0 0.069-1.0 0.069-1.0 0.4-0.01 0.4-0.01 0.4-0.01 0.4-0.01 0.4-0.01 0.4-0.01 0.4-0.01 0.4-0.01 0.4-0.01 0.4-0.01
    0 395.062347 542.019287 259.171783 378.006470 672.683350 422.525299 435.934174 535.840027 341.197662 546.919556 1194.309570 1331.900391 1614.625854 1096.244019 5350.819336 1109.356201 1231.328979 2721.381592 1671.956787 1572.913452
    1 395.732605 676.451477 263.082733 365.738861 672.841858 436.986450 408.519165 578.790833 357.324097 518.666321 1226.035278 1348.827026 1727.460327 1040.826294 5548.753418 1241.292969 1197.370972 2785.768066 1749.072510 1544.390259
    2 385.598785 565.850403 258.483124 350.939362 680.929993 416.969391 408.744873 572.143921 350.399933 546.654846 1236.471436 1323.095703 1817.804810 949.081665 5418.844727 1285.566650 1229.268799 2961.105225 1643.184204 1486.991211
    3 375.055664 569.737793 252.117035 364.043427 651.953247 449.630371 390.186218 521.915283 366.314545 534.204285 1192.580566 1219.429932 1844.903687 888.757324 5756.974121 1303.747803 1190.227539 3127.305664 1713.993042 1427.290771
    4 400.554443 590.014893 241.840240 376.104645 681.107056 466.177185 380.395752 531.094727 370.980286 537.930359 1168.627441 1194.065186 1756.768799 887.986389 5481.615234 1324.906250 1268.030762 2997.608887 1744.911865 1426.806763
    16379 433.562714 624.462646 643.004944 518.733643 563.566589 578.520813 348.858429 330.541473 9484.205078 376.017700 1352.349854 1400.204102 2025.719604 934.456665 2333.292725 1499.182251 1344.230347 1172.255371 1347.495239 756.805908
    16380 462.284454 616.137512 597.266296 487.652924 572.072327 612.569275 328.044495 331.003693 8237.546875 373.477081 1305.663696 1453.817993 1847.917114 1012.087402 2349.776611 1498.571411 1446.490479 1191.984253 1482.415894 712.499878
    16381 472.551483 612.926758 615.009460 516.941528 579.562378 624.847717 308.531097 308.009369 2722.457275 352.414612 1384.178711 1428.226440 1641.537109 975.000366 2291.302734 1541.471436 1334.644897 1173.113770 1520.151367 587.645203
    16382 486.679413 637.962769 616.344116 502.372345 593.559937 673.262634 307.834229 322.522400 2823.112305 336.442596 1258.534058 1423.324341 1560.817139 1023.877014 2185.760742 1455.700928 1387.281250 1124.065552 1572.194336 618.202820
    16383 489.893646 614.733704 614.638000 511.408234 595.268982 673.656921 318.466736 305.981110 1768.038330 361.107300 1114.534912 1386.146484 1548.830078 1009.011658 2117.508789 1569.905518 1396.511353 1070.131104 1602.530029 654.377380

    16384 rows × 3000 columns

    dataset_test, num_test_examples = ppd.tfds_from_pddf(
        features_df=sim_dirty, labels_df=sim_labbool, frac_val=False)
    dataset_test
    
    2022-08-23 13:50:29.331909: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2022-08-23 13:50:29.331967: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
    2022-08-23 13:50:29.331995: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node100): /proc/driver/nvidia/version does not exist
    2022-08-23 13:50:29.332353: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    <MapDataset element_spec=(TensorSpec(shape=(16384, 1), dtype=tf.float32, name=None), TensorSpec(shape=(16384, 1), dtype=tf.float32, name=None))>
    
  • now apply each model to the held-out test dataset and evaluate prediction performance. We also get info on the inference time in total and for each step.
    eva = pd.DataFrame()
    for i in model_ls:
        logged_scaler = Path(f'/beegfs/ye53nis/drmed-git/data/mlruns/10/{i}/params/scaler')
        logged_scaler = !cat $logged_scaler
        logged_scaler = logged_scaler[0]
    
        logged_model = Path(f'/beegfs/ye53nis/drmed-git/data/mlruns/10/{i}/artifacts/model')
        logged_model = load_model(logged_model, compile=False)
        logged_model.compile(loss=bm.binary_ce_dice_loss(),
                             optimizer=Adam(),
                             metrics = bm.unet_metrics([0.1, 0.3, 0.5, 0.7, 0.9]))
    
        dataset = dataset_test.map(
            lambda trace, label: ppd.tfds_scale_trace_and_label(trace, label, logged_scaler),
            num_parallel_calls=tf.data.AUTOTUNE)
        dataset = dataset.map(
            ppd.tfds_pad_trace_and_label,
            num_parallel_calls=tf.data.AUTOTUNE)
        dataset = dataset.batch(1)
        print(f'model: {i}')
        eva_new = logged_model.evaluate(dataset, verbose=2, return_dict=True)
        eva_new = pd.DataFrame(eva_new.values(), index=eva_new.keys(), columns=[f'{i}'])
        eva = pd.concat([eva, eva_new], axis='columns')
        print('----------------------------------------')
    
    
    
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    model: ff67be0b68e540a9a29a36a2d0c7a5be
    3000/3000 - 47s - loss: 0.1929 - tp0.1: 7216054.0000 - fp0.1: 1445349.0000 - tn0.1: 40150456.0000 - fn0.1: 340179.0000 - precision0.1: 0.8331 - recall0.1: 0.9550 - tp0.3: 7068737.0000 - fp0.3: 874006.0000 - tn0.3: 40721760.0000 - fn0.3: 487496.0000 - precision0.3: 0.8900 - recall0.3: 0.9355 - tp0.5: 6879515.0000 - fp0.5: 517256.0000 - tn0.5: 41078488.0000 - fn0.5: 676718.0000 - precision0.5: 0.9301 - recall0.5: 0.9104 - tp0.7: 6640601.0000 - fp0.7: 285835.0000 - tn0.7: 41309920.0000 - fn0.7: 915632.0000 - precision0.7: 0.9587 - recall0.7: 0.8788 - tp0.9: 6140071.0000 - fp0.9: 88637.0000 - tn0.9: 41507120.0000 - fn0.9: 1416162.0000 - precision0.9: 0.9858 - recall0.9: 0.8126 - accuracy: 0.9757 - auc: 0.9756 - f1: 0.9202 - 47s/epoch - 16ms/step
    ----------------------------------------
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    model: 347669d050f344ad9fb9e480c814f727
    
  • we get run1_2_my with a summary of the best hyperparameter training runs from the code block #+CALL: get-hparam-comparison(). This assigns among others the variables run1_2_my, cond2, cond3
    Run ID: valauc valrecall0.5 valprecision0.5 hpbatchsize hpfirstfilters hpinputsize hplrpower hplrstart hpnlevels hppoolsize hpscaler
    (9051e32b87d84f3485b980067addec30, 61ff87bdb89b4e2ba64f8dacc774992d) 0.981 0.8975 0.918 26 44 16384 1 0.0136170138242663 7 2 standard
    (93b168c0ff7942c8a908a94129daf973, f243b3b742de4dbcb7ccfbd4244706f8) 0.976 0.893 0.852 15 23 16384 7 0.0305060808685107 6 4 quantg
    (a5b8551144ff46e697a39cd1551e1475, 98cf8cdef9c54b5286e277e75e2ab8c1) 0.984 0.916 0.909 20 78 16384 4 0.0584071108418767 4 4 standard
    (00f2635d9fa2463c9a066722163405be, d0a8e1748b194f3290d471b6b44f19f8) 0.987 0.929 0.9065 28 6 16384 1 0.0553313915596308 5 4 minmax
    (5604d43c1ece461b8e6eaa0dfb65d6dc, 3612536a77f34f22bc83d1d809140aa6) 0.9745 0.885 0.8985 20 128 16384 1 0.043549707353273 3 4 standard
    (7cafab027cdd4fc9bf20a43e989df510, 16dff15d935f45e2a836b1f41b07b4e3) 0.978 0.8905 0.891 10 16 8192 1 0.0627676336651573 5 4 robust
    (0e328920e86049928202db95e8cfb7be, bf9d2725eb16462d9a101f0a077ce2b5) 0.976 0.875 0.888 14 16 16384 5 0.0192390310290551 9 2 robust
    (1c954fbc02b747bc813c587ac703c74a, ba49a80c2616407a8f1fe1fd12096fe0) 0.962 0.856 0.8585 17 16 16384 5 0.0101590069352232 3 4 l2
    (3cbd945b62ec4634839372e403f6f377, 458b36a70db843719d202a8eda448f17) 0.972 0.872 0.9135 9 64 4096 1 0.0100697459464075 5 4 maxabs
  • now get hparams and metrics from final trained models. Inference times and model sizes are transferred by hand from the evaluation results.
    model_dict = {
        ('9051e32b87d84f3485b980067addec30', '61ff87bdb89b4e2ba64f8dacc774992d') :  '484af471c61943fa90e5f78e78a229f0',
        ('93b168c0ff7942c8a908a94129daf973', 'f243b3b742de4dbcb7ccfbd4244706f8') :  '0cd2023eeaf745aca0d3e8ad5e1fc653',
        ('a5b8551144ff46e697a39cd1551e1475', '98cf8cdef9c54b5286e277e75e2ab8c1') :  'fe81d71c52404ed790b3a32051258da9',
        ('00f2635d9fa2463c9a066722163405be', 'd0a8e1748b194f3290d471b6b44f19f8') :  'ff67be0b68e540a9a29a36a2d0c7a5be',
        ('5604d43c1ece461b8e6eaa0dfb65d6dc', '3612536a77f34f22bc83d1d809140aa6') :  '19e3e786e1bc4e2b93856f5dc9de8216',
        ('7cafab027cdd4fc9bf20a43e989df510', '16dff15d935f45e2a836b1f41b07b4e3') :  '347669d050f344ad9fb9e480c814f727',
        ('0e328920e86049928202db95e8cfb7be', 'bf9d2725eb16462d9a101f0a077ce2b5') :  'c1204e3a8a1e4c40a35b5b7b1922d1ce',
        ('1c954fbc02b747bc813c587ac703c74a', 'ba49a80c2616407a8f1fe1fd12096fe0') :  '714af8cd12c1441eac4ca980e8c20070',
        ('3cbd945b62ec4634839372e403f6f377', '458b36a70db843719d202a8eda448f17') :  '34a6d207ac594035b1009c330fb67a65'}
    
    
    inference = pd.DataFrame([['42s', '14ms', '275 MB'],
                              ['65s', '22ms', '200 MB'],
                              ['201s', '67ms', '186 MB'],
                              ['45s', '15ms', '14 MB'],
                              ['370s', '123ms', '172 MB'],
                              ['95s', '32ms', '73 MB'],
                              ['230s', '77ms', '312 MB'],
                              ['361s', '120ms', '234 MB'],
                              ['230s', '77ms', '7 MB']],
                              columns=['HPC inference time (whole test dataset)', 'HPC inference time (one trace)', 'Model size'],
                             index=['484af471c61943fa90e5f78e78a229f0',
                                    '0cd2023eeaf745aca0d3e8ad5e1fc653',
                                    'fe81d71c52404ed790b3a32051258da9',
                                    'ff67be0b68e540a9a29a36a2d0c7a5be',
                                    '19e3e786e1bc4e2b93856f5dc9de8216',
                                    '347669d050f344ad9fb9e480c814f727',
                                    'c1204e3a8a1e4c40a35b5b7b1922d1ce',
                                    '714af8cd12c1441eac4ca980e8c20070',
                                    '34a6d207ac594035b1009c330fb67a65']).T
    
    evaluation = pd.concat([eva, inference], axis='index')
    
    index = [f'test {e}' for e in evaluation.index]
    evaluation.index = index
    training = run1_2_my.loc[cond2 & cond3]
    
    training = training.rename(index=model_dict).T
    final = pd.concat([training, evaluation])
    final.to_csv('data/exp-220227-unet/mlflow/2022-08-23_all-models.csv')
    
      484af471c61943fa90e5f78e78a229f0 0cd2023eeaf745aca0d3e8ad5e1fc653 fe81d71c52404ed790b3a32051258da9 ff67be0b68e540a9a29a36a2d0c7a5be 19e3e786e1bc4e2b93856f5dc9de8216 347669d050f344ad9fb9e480c814f727 c1204e3a8a1e4c40a35b5b7b1922d1ce 714af8cd12c1441eac4ca980e8c20070 34a6d207ac594035b1009c330fb67a65
    valauc 0.981 0.976 0.984 0.987 0.9745 0.978 0.976 0.962 0.972
    valrecall0.5 0.8975 0.893 0.916 0.929 0.885 0.8905 0.875 0.856 0.872
    valprecision0.5 0.918 0.852 0.909 0.9065 0.8985 0.891 0.888 0.8585 0.9135
    hpbatchsize 26 15 20 28 20 10 14 17 9
    hpfirstfilters 44 23 78 6 128 16 16 16 64
    hpinputsize 16384 16384 16384 16384 16384 8192 16384 16384 4096
    hplrpower 1 7 4 1 1 1 5 5 1
    hplrstart 0.0136170138242663 0.0305060808685107 0.0584071108418767 0.0553313915596308 0.043549707353273 0.0627676336651573 0.0192390310290551 0.0101590069352232 0.0100697459464075
    hpnlevels 7 6 4 5 3 5 9 3 5
    hppoolsize 2 4 4 4 4 4 2 4 4
    hpscaler standard quantg standard minmax standard robust robust l2 maxabs
    test loss 0.228701 0.291168 0.207543 0.192903 0.39377 0.289283 0.264141 0.18674 0.318487
    test tp0.1 7120162.0 7030956.0 7195616.0 7216054.0 6702946.0 6968717.0 7094671.0 7191122.0 6898118.0
    test fp0.1 1418863.0 1944947.0 1635920.0 1445349.0 1573303.0 1346502.0 1783490.0 1089514.0 1865476.0
    test tn0.1 40176900.0 39650840.0 39959820.0 40150456.0 40022456.0 40249248.0 39812172.0 40506272.0 39730280.0
    test fn0.1 436071.0 525277.0 360617.0 340179.0 853287.0 587516.0 461562.0 365111.0 658115.0
    test precision0.1 0.833838 0.783315 0.814764 0.833128 0.809901 0.838068 0.799115 0.868426 0.787133
    test recall0.1 0.94229 0.930484 0.952276 0.95498 0.887075 0.922248 0.938916 0.951681 0.912904
    test tp0.3 6967838.0 6780626.0 7057588.0 7068737.0 6536047.0 6749402.0 6893942.0 7062843.0 6565790.0
    test fp0.3 864351.0 1063352.0 1032334.0 874006.0 1023316.0 704964.0 923800.0 687977.0 850301.0
    test tn0.3 40731400.0 40532408.0 40563420.0 40721760.0 40572400.0 40890776.0 40671944.0 40907784.0 40745504.0
    test fn0.3 588395.0 775607.0 498645.0 487496.0 1020186.0 806831.0 662291.0 493390.0 990443.0
    test precision0.3 0.889641 0.864437 0.872393 0.889962 0.864629 0.905429 0.881833 0.911238 0.885344
    test recall0.3 0.922131 0.897355 0.934009 0.935484 0.864987 0.893223 0.912352 0.934704 0.868924
    test tp0.5 6806719.0 6547596.0 6873409.0 6879515.0 6381884.0 6514890.0 6657803.0 6907934.0 6236955.0
    test fp0.5 561522.0 671113.0 627857.0 517256.0 718562.0 394738.0 506659.0 432424.0 431128.0
    test tn0.5 41034252.0 40924664.0 40967908.0 41078488.0 40877184.0 41201060.0 41089052.0 41163320.0 41164680.0
    test fn0.5 749514.0 1008637.0 682824.0 676718.0 1174349.0 1041343.0 898430.0 648299.0 1319278.0
    test precision0.5 0.923792 0.907031 0.9163 0.93007 0.8988 0.942871 0.929282 0.94109 0.935345
    test recall0.5 0.900809 0.866516 0.909634 0.910442 0.844585 0.862188 0.881101 0.914203 0.825405
    test tp0.7 6576475.0 6228335.0 6630207.0 6640601.0 6136998.0 6185431.0 6321268.0 6663553.0 5823676.0
    test fp0.7 327204.0 381447.0 350175.0 285835.0 446064.0 178898.0 240152.0 227729.0 194655.0
    test tn0.7 41268520.0 41214264.0 41245624.0 41309920.0 41149748.0 41416908.0 41355624.0 41368048.0 41401160.0
    test fn0.7 979758.0 1327898.0 926026.0 915632.0 1419235.0 1370802.0 1234965.0 892680.0 1732557.0
    test precision0.7 0.952604 0.942291 0.949834 0.958733 0.932241 0.971891 0.963399 0.966954 0.967656
    test recall0.7 0.870338 0.824265 0.877449 0.878824 0.812177 0.818587 0.836563 0.881862 0.770712
    test tp0.9 6102547.0 5564160.0 6120144.0 6140071.0 5641558.0 5650455.0 5713166.0 6190425.0 5096155.0
    test fp0.9 119229.0 132517.0 115037.0 88637.0 190273.0 48800.0 64046.0 69867.0 44052.0
    test tn0.9 41476596.0 41463236.0 41480668.0 41507120.0 41405512.0 41546992.0 41531720.0 41525924.0 41551772.0
    test fn0.9 1453686.0 1992073.0 1436089.0 1416162.0 1914675.0 1905778.0 1843067.0 1365808.0 2460078.0
    test precision0.9 0.980837 0.976738 0.98155 0.98577 0.967373 0.991437 0.988914 0.98884 0.99143
    test recall0.9 0.807618 0.736367 0.809946 0.812584 0.74661 0.747787 0.756087 0.819247 0.674431
    test accuracy 0.973328 0.965825 0.973334 0.975708 0.961488 0.970784 0.971414 0.978012 0.964389
    test auc 0.968705 0.961385 0.973758 0.975623 0.939052 0.95903 0.966802 0.974297 0.953582
    test f1 0.912155 0.886311 0.912955 0.920152 0.87085 0.900726 0.90455 0.927452 0.876943
    test HPC inference time (whole test dataset) 42s 65s 201s 45s 370s 95s 230s 361s 230s
    test HPC inference time (one trace) 14ms 22ms 67ms 15ms 123ms 32ms 77ms 120ms 77ms
    test Model size 275 MB 200 MB 186 MB 14 MB 172 MB 73 MB 312 MB 234 MB 7 MB
  • for the final data we only take recall, precision, and f1 at a 0.5 prediction threshold.
    final.loc[['val_auc', 'test auc', 'val_recall0.5', 'test recall0.5', 'val_precision0.5',
               'test precision0.5', 'test f1', 'hp_batch_size', 'hp_input_size', 'hp_lr_power',
               'hp_lr_start', 'hp_n_levels', 'hp_pool_size', 'hp_scaler',
               'test HPC inference time (whole test dataset)', 'test HPC inference time (one trace)',
               'test Model size']]
    
      0cd2023eeaf745aca0d3e8ad5e1fc653 ff67be0b68e540a9a29a36a2d0c7a5be 484af471c61943fa90e5f78e78a229f0 fe81d71c52404ed790b3a32051258da9 19e3e786e1bc4e2b93856f5dc9de8216 347669d050f344ad9fb9e480c814f727 c1204e3a8a1e4c40a35b5b7b1922d1ce 714af8cd12c1441eac4ca980e8c20070 34a6d207ac594035b1009c330fb67a65
    valauc 0.976 0.987 0.981 0.984 0.9745 0.978 0.976 0.962 0.972
    test auc 0.961385 0.975623 0.968705 0.973758 0.939052 0.95903 0.966802 0.974297 0.953582
    valrecall0.5 0.893 0.929 0.8975 0.916 0.885 0.8905 0.875 0.856 0.872
    test recall0.5 0.866516 0.910442 0.900809 0.909634 0.844585 0.862188 0.881101 0.914203 0.825405
    valprecision0.5 0.852 0.9065 0.918 0.909 0.8985 0.891 0.888 0.8585 0.9135
    test precision0.5 0.907031 0.93007 0.923792 0.9163 0.8988 0.942871 0.929282 0.94109 0.935345
    test f1 0.886311 0.920152 0.912155 0.912955 0.87085 0.900726 0.90455 0.927452 0.876943
    hpbatchsize 15 28 26 20 20 10 14 17 9
    hpinputsize 16384 16384 16384 16384 16384 8192 16384 16384 4096
    hplrpower 7 1 1 4 1 1 5 5 1
    hplrstart 0.0305060808685107 0.0553313915596308 0.0136170138242663 0.0584071108418767 0.043549707353273 0.0627676336651573 0.0192390310290551 0.0101590069352232 0.0100697459464075
    hpnlevels 6 5 7 4 3 5 9 3 5
    hppoolsize 4 4 2 4 4 4 2 4 4
    hpscaler quantg minmax standard standard standard robust robust l2 maxabs
    test HPC inference time (whole test dataset) 65s 45s 42s 201s 370s 95s 230s 361s 230s
    test HPC inference time (one trace) 22ms 15ms 14ms 67ms 123ms 32ms 77ms 120ms 77ms
    test Model size 200 MB 14 MB 275 MB 186 MB 172 MB 73 MB 312 MB 234 MB 7 MB
  • load held-out test data from simulation.
    col_per_example = 3
    lab_thresh = 0.04
    artifact = 0
    model_type = 1
    fwhm = 250
    
    output_path = "/beegfs/ye53nis/drmed-git/data/exp-220227-unet/2022-04-25_simulations/"
    sim_path = Path('/beegfs/ye53nis/saves/firstartifact_Nov2020_test')
    
    sim, _, nsamples, sim_params = isfc.import_from_csv(
        folder=sim_path,
        header=12,
        frac_train=1,
        col_per_example=col_per_example,
        dropindex=None,
        dropcolumns=None)
    
    diffrates = sim_params.loc['diffusion rate of molecules in micrometer^2 / s'].astype(np.float32)
    nmols = sim_params.loc['number of fast molecules'].astype(np.float32)
    clusters = sim_params.loc['diffusion rate of clusters in micrometer^2 / s'].astype(np.float32)
    sim_columns = [f'{d:.4}-{c:.4}' for d, c in zip(
        np.repeat(diffrates, nsamples[0]),
        np.repeat(clusters, nsamples[0]))]
    sim_columns_idx = ['50.0-0.01', '50.0-0.1', '50.0-1.0',
                       '0.2-0.01', '0.2-0.1', '0.2-1.0',
                       '0.069-0.01', '0.069-0.1', '0.069-1.0']
    sim_columns_txt = ['fast molecules and slow clusters:\nsimulations',
                       'fast molecules and medium clusters:\nsimulations',
                       'fast molecules and fast clusters:\nsimulations',
                       'medium molecules and slow clusters:\nsimulations',
                       'medium molecules and medium clusters:\nsimulations',
                       'medium molecules and fast clusters:\nsimulations',
                       'slow molecules and slow clusters:\nsimulations',
                       'slow molecules and medium clusters:\nsimulations',
                       'slow molecules and fast clusters:\nsimulations']
    
    sim_sep = isfc.separate_data_and_labels(array=sim,
                                            nsamples=nsamples,
                                            col_per_example=col_per_example)
    sim_dirty = sim_sep['0']
    sim_dirty.columns = sim_columns
    
    sim_labels = sim_sep['1']
    sim_labels.columns = sim_columns
    sim_labbool = sim_labels > lab_thresh
    sim_labbool.columns = sim_columns
    sim_clean = sim_sep['2']
    sim_clean.columns = sim_columns
    
    sim_dirty
    
    2022-04-25 11:32:12,435 - 1/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.069/1.0/traces_brightclust_Nov2020_D0.069_set010.csv
    2022-04-25 11:32:15,346 - 2/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/50/0.1/traces_brightclust_Nov2020_D50_set003.csv
    2022-04-25 11:32:18,091 - 3/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.4/0.1/traces_brightclust_Nov2020_D0.4_set001.csv
    2022-04-25 11:32:21,085 - 4/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.2/0.1/traces_brightclust_Nov2020_D0.2_set007.csv
    2022-04-25 11:32:23,781 - 5/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/3.0/1.0/traces_brightclust_Nov2020_D3.0_set002.csv
    2022-04-25 11:32:26,490 - 6/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/3.0/0.01/traces_brightclust_Nov2020_D3.0_set004.csv
    2022-04-25 11:32:29,216 - 7/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/50/0.01/traces_brightclust_Nov2020_D50_set002.csv
    2022-04-25 11:32:32,164 - 8/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.2/1.0/traces_brightclust_Nov2020_D0.2_set005.csv
    2022-04-25 11:32:35,219 - 9/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.6/1.0/traces_brightclust_Nov2020_D0.6_set009.csv
    2022-04-25 11:32:38,070 - 10/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/10/0.1/traces_brightclust_Nov2020_D10_set001.csv2022-04-25 11:32:40,862 - 11/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.08/1.0/traces_brightclust_Nov2020_D0.08_set001.csv
    2022-04-25 11:32:44,030 - 12/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.6/0.1/traces_brightclust_Nov2020_D0.6_set003.csv
    2022-04-25 11:32:46,779 - 13/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.1/1.0/traces_brightclust_Nov2020_D0.1_set001.csv
    2022-04-25 11:32:49,564 - 14/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.4/1.0/traces_brightclust_Nov2020_D0.4_set005.csv
    2022-04-25 11:32:52,553 - 15/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/10/1.0/traces_brightclust_Nov2020_D10_set005.csv
    2022-04-25 11:32:55,473 - 16/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/1.0/1.0/traces_brightclust_Nov2020_D1.0_set005.csv
    2022-04-25 11:32:58,258 - 17/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.069/0.1/traces_brightclust_Nov2020_D0.069_set001.csv
    2022-04-25 11:33:01,043 - 18/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/50/1.0/traces_brightclust_Nov2020_D50_set001.csv
    2022-04-25 11:33:03,986 - 19/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.1/0.01/traces_brightclust_Nov2020_D0.1_set002.csv
    2022-04-25 11:33:06,905 - 20/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.08/0.1/traces_brightclust_Nov2020_D0.08_set003.csv
    2022-04-25 11:33:10,236 - 21/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/1.0/0.01/traces_brightclust_Nov2020_D1.0_set006.csv
    2022-04-25 11:33:12,972 - 22/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/1.0/0.1/traces_brightclust_Nov2020_D1.0_set003.csv
    2022-04-25 11:33:15,879 - 23/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.2/0.01/traces_brightclust_Nov2020_D0.2_set002.csv
    2022-04-25 11:33:18,677 - 24/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.1/0.1/traces_brightclust_Nov2020_D0.1_set005.csv
    2022-04-25 11:33:21,515 - 25/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/3.0/0.1/traces_brightclust_Nov2020_D3.0_set007.csv
    2022-04-25 11:33:24,287 - 26/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.08/0.01/traces_brightclust_Nov2020_D0.08_set005.csv
    2022-04-25 11:33:27,293 - 27/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.069/0.01/traces_brightclust_Nov2020_D0.069_set005.csv
    2022-04-25 11:33:30,137 - 28/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/10/0.01/traces_brightclust_Nov2020_D10_set002.csv
    2022-04-25 11:33:32,952 - 29/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.6/0.01/traces_brightclust_Nov2020_D0.6_set008.csv
    2022-04-25 11:33:35,872 - 30/30: /beegfs/ye53nis/saves/firstartifact_Nov2020_test/0.4/0.01/traces_brightclust_Nov2020_D0.4_set008.csv
    2022-04-25 11:33:36,060 - The given DataFrame was split into 3 parts with shapes: [(16384, 3000), (16384, 3000), (16384, 3000)]
    
      0.069-1.0 0.069-1.0 0.069-1.0 0.069-1.0 0.069-1.0 0.069-1.0 0.069-1.0 0.069-1.0 0.069-1.0 0.069-1.0 0.4-0.01 0.4-0.01 0.4-0.01 0.4-0.01 0.4-0.01 0.4-0.01 0.4-0.01 0.4-0.01 0.4-0.01 0.4-0.01
    0 395.062347 542.019287 259.171783 378.006470 672.683350 422.525299 435.934174 535.840027 341.197662 546.919556 1194.309570 1331.900391 1614.625854 1096.244019 5350.819336 1109.356201 1231.328979 2721.381592 1671.956787 1572.913452
    1 395.732605 676.451477 263.082733 365.738861 672.841858 436.986450 408.519165 578.790833 357.324097 518.666321 1226.035278 1348.827026 1727.460327 1040.826294 5548.753418 1241.292969 1197.370972 2785.768066 1749.072510 1544.390259
    2 385.598785 565.850403 258.483124 350.939362 680.929993 416.969391 408.744873 572.143921 350.399933 546.654846 1236.471436 1323.095703 1817.804810 949.081665 5418.844727 1285.566650 1229.268799 2961.105225 1643.184204 1486.991211
    3 375.055664 569.737793 252.117035 364.043427 651.953247 449.630371 390.186218 521.915283 366.314545 534.204285 1192.580566 1219.429932 1844.903687 888.757324 5756.974121 1303.747803 1190.227539 3127.305664 1713.993042 1427.290771
    4 400.554443 590.014893 241.840240 376.104645 681.107056 466.177185 380.395752 531.094727 370.980286 537.930359 1168.627441 1194.065186 1756.768799 887.986389 5481.615234 1324.906250 1268.030762 2997.608887 1744.911865 1426.806763
    16379 433.562714 624.462646 643.004944 518.733643 563.566589 578.520813 348.858429 330.541473 9484.205078 376.017700 1352.349854 1400.204102 2025.719604 934.456665 2333.292725 1499.182251 1344.230347 1172.255371 1347.495239 756.805908
    16380 462.284454 616.137512 597.266296 487.652924 572.072327 612.569275 328.044495 331.003693 8237.546875 373.477081 1305.663696 1453.817993 1847.917114 1012.087402 2349.776611 1498.571411 1446.490479 1191.984253 1482.415894 712.499878
    16381 472.551483 612.926758 615.009460 516.941528 579.562378 624.847717 308.531097 308.009369 2722.457275 352.414612 1384.178711 1428.226440 1641.537109 975.000366 2291.302734 1541.471436 1334.644897 1173.113770 1520.151367 587.645203
    16382 486.679413 637.962769 616.344116 502.372345 593.559937 673.262634 307.834229 322.522400 2823.112305 336.442596 1258.534058 1423.324341 1560.817139 1023.877014 2185.760742 1455.700928 1387.281250 1124.065552 1572.194336 618.202820
    16383 489.893646 614.733704 614.638000 511.408234 595.268982 673.656921 318.466736 305.981110 1768.038330 361.107300 1114.534912 1386.146484 1548.830078 1009.011658 2117.508789 1569.905518 1396.511353 1070.131104 1602.530029 654.377380

    16384 rows × 3000 columns

  • save correlations before correction, then load each model, predict artifacts, and correct it with the delete and shift method, which we later renamed cut and stitch, then fit the correlations with Dominic Waithe's focuspoint
    # before correction
    correlate.correlate_timetrace_and_save(
        df=sim_dirty,
        out_path="/beegfs/ye53nis/drmed-git/data/exp-220227-unet/2022-04-25_simulations/",
        txt='dirty')
    
    for mid, mname in enumerate(model_name_ls):
        logged_scaler = Path(f'/beegfs/ye53nis/drmed-git/data/mlruns/10/{model_ls[model_id]}/params/scaler')
        logged_scaler = !cat $logged_scaler
        logged_scaler = logged_scaler[0]
    
        logged_model = Path(f'/beegfs/ye53nis/drmed-git/data/mlruns/10/{model_ls[model_id]}/artifacts/model')
        logged_model = load_model(logged_model, compile=False)
        logged_model.compile(loss=bm.binary_ce_dice_loss(),
                             optimizer=Adam(),
                             metrics = bm.unet_metrics([0.1, 0.3, 0.5, 0.7, 0.9]))
    
        ans.predict_correct_correlate_simulations(
            sim_df=sim_dirty,
            model=logged_model,
            scaler=logged_scaler,
            out_path="/beegfs/ye53nis/drmed-git/data/exp-220227-unet/2022-04-25_simulations/",
            txt=mname)
    
    
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    
  • we then fitted the data with Dominic Waithe's focuspoint.
  • One thing I realized later, is that fit results from all curves with simulated diffusion coefficients of \(10\) and \(50 \mu m^2/s\) were not comparable, because the time step size of \(1ms\) of these traces was not fine enough (\(10\mu m^2/s \propto 1.1ms\)). It was still useful to simulate them for training the model.
  • for the final paper, we only compared three different times: \(\{0.069, 0.2, 3.0\}\mu m^2 s^{-1}\)

2.6.11 Analysis 2: correct simulated test data

2.6.11.1 illustrative plots
  • call #+CALL: jupyter-set-output-directory()
    ./data/exp-220227-unet/jupyter
    
  • load modules
    %cd ~/Programme/drmed-git
    
    import logging
    import os
    import sys
    
    import matplotlib.pyplot as plt
    import numpy as np
    import pandas as pd
    import seaborn as sns
    
    from mlflow.keras import load_model
    from pathlib import Path
    from pprint import pprint
    from sklearn.preprocessing import MaxAbsScaler
    from tensorflow.keras.optimizers import Adam
    
    
    
    FLUOTRACIFY_PATH = '/home/lex/Programme/drmed-git/src/'
    sys.path.append(FLUOTRACIFY_PATH)
    from fluotracify.applications import corr_fit_object as cfo
    from fluotracify.imports import ptu_utils as ptu
    from fluotracify.training import (build_model as bm,
                                      preprocess_data as ppd)
    from fluotracify.simulations import (
       import_simulation_from_csv as isfc,
       analyze_simulations as ans,
    )
    
    logging.basicConfig(filename="/home/lex/Programme/drmed-git/data/exp-220227-unet/jupyter.log",
                        filemode='w', format='%(asctime)s - %(message)s',
                        force=True,
                        level=logging.DEBUG)
    
    sns.set_theme(style="whitegrid", font_scale=2, palette='colorblind',
                  context='paper')
    
    model_ls = ['ff67be0b68e540a9a29a36a2d0c7a5be', '347669d050f344ad9fb9e480c814f727',
                '714af8cd12c1441eac4ca980e8c20070', '34a6d207ac594035b1009c330fb67a65',
                '484af471c61943fa90e5f78e78a229f0', '0cd2023eeaf745aca0d3e8ad5e1fc653',
                'fe81d71c52404ed790b3a32051258da9', '19e3e786e1bc4e2b93856f5dc9de8216',
                'c1204e3a8a1e4c40a35b5b7b1922d1ce']
    
    model_name_ls = [f'{s:.5}' for s in model_ls]
    
    pred_thresh = 0.5
    
    
    /home/lex/Programme/drmed-git
    
  • load simulated data
    col_per_example = 3
    lab_thresh = 0.04
    artifact = 0
    model_type = 1
    fwhm = 250
    sim_path = Path('/home/lex/Programme/drmed-collections/drmed-simexps/firstartifact_Nov2020_test')
    
    sim, _, nsamples, sim_params = isfc.import_from_csv(
        folder=sim_path,
        header=12,
        frac_train=1,
        col_per_example=col_per_example,
        dropindex=None,
        dropcolumns=None)
    
    diffrates = sim_params.loc['diffusion rate of molecules in micrometer^2 / s'].astype(np.float32)
    nmols = sim_params.loc['number of fast molecules'].astype(np.float32)
    clusters = sim_params.loc['diffusion rate of clusters in micrometer^2 / s'].astype(np.float32)
    sim_columns = [f'{d:.4}-{c:.4}' for d, c in zip(
        np.repeat(diffrates, nsamples[0]),
        np.repeat(clusters, nsamples[0]))]
    
    sim_sep = isfc.separate_data_and_labels(array=sim,
                                            nsamples=nsamples,
                                            col_per_example=col_per_example)
    sim_dirty = sim_sep['0']
    sim_dirty.columns = sim_columns
    
    sim_labels = sim_sep['1']
    sim_labels.columns = sim_columns
    sim_labbool = sim_labels > lab_thresh
    sim_labbool.columns = sim_columns
    sim_clean = sim_sep['2']
    sim_clean.columns = sim_columns
    
    sim_dirty
    
      0.069-0.1 0.069-0.1 0.069-0.1 0.069-0.1 0.069-0.1 0.069-0.1 0.069-0.1 0.069-0.1 0.069-0.1 0.069-0.1 50.0-0.01 50.0-0.01 50.0-0.01 50.0-0.01 50.0-0.01 50.0-0.01 50.0-0.01 50.0-0.01 50.0-0.01 50.0-0.01
    0 1187.467896 907.677734 480.048798 454.669403 466.063232 384.734467 543.981323 640.921509 795.946167 410.471893 1897.398193 2279.646484 3088.531006 2034.432495 2187.548584 2105.736084 1789.366577 2023.093994 2331.185791 2185.028076
    1 1184.055176 945.760315 471.065216 458.392487 473.306152 395.165527 558.603088 622.421204 776.199402 409.149170 1499.969849 2199.652100 3207.333008 1650.523926 2122.935791 2791.281006 1661.286377 1111.879761 1853.699585 1926.844971
    2 1191.848877 980.117798 459.479706 426.087982 482.370209 425.123413 551.536072 624.498535 778.671265 400.971954 1822.985229 2456.422607 2969.562500 1934.118286 1457.731812 2251.077393 1903.003662 2063.915527 2198.018066 2038.683105
    3 1199.065918 974.313110 462.205566 444.041809 463.703125 434.186615 573.044128 626.252502 747.284058 393.178162 1741.839355 2467.149414 2588.980957 2136.627686 1930.263672 2323.700684 2133.313721 1638.169312 1926.058716 1815.259521
    4 1221.957397 968.779175 464.918030 455.205292 474.615540 437.029419 586.489136 619.319092 781.954102 406.468018 2431.400879 2246.336670 3000.961182 1915.518066 2052.773682 2359.145508 1699.926147 1862.709595 2291.338379 1332.422241
    16379 506.409668 1012.403931 855.006226 674.470703 769.859192 2110.732178 799.565247 1981.221436 528.844604 483.055878 1512.586548 3212.712891 1491.119995 1843.866943 1748.956665 2048.602051 1662.244385 2593.879639 1921.427856 1664.831909
    16380 536.809692 1022.029724 840.287720 671.095215 738.908997 2118.984863 807.995789 2624.343262 552.687012 479.768372 1661.331055 3190.809570 1770.193970 2081.854248 2164.372803 2295.646729 1846.683594 2038.272339 2222.708252 2122.753662
    16381 570.668884 989.891235 839.180298 689.863586 695.739136 2033.681885 786.547852 3528.163574 572.166077 484.491211 1643.470337 2564.206787 2025.219971 2104.706787 1792.828613 2106.199463 2087.914062 1457.817871 1874.736938 1683.072021
    16382 562.505310 977.029785 1005.927673 683.250183 661.608337 2566.123047 805.594116 3731.086426 566.710571 489.289673 1556.492188 2783.619385 1312.174561 2378.643311 2466.965576 2160.641357 1691.332520 2013.095093 1632.475708 1352.443237
    16383 567.307373 1006.794067 982.376526 677.099854 657.040588 2545.080322 784.917969 3850.334717 570.241699 512.688232 2127.414551 2448.062012 1398.359253 1665.321167 2241.687256 1823.699829 1340.686035 1972.661743 1550.770142 1727.808228

    16384 rows × 1500 columns

  • define plotting functions
    plot1_index = ['3.0-0.01', '3.0-0.1', '3.0-1.0',
                   '0.2-0.01', '0.2-0.1', '0.2-1.0',
                   '0.069-0.01', '0.069-0.1', '0.069-1.0']
    
    plot1_traceno = [1, 1, 0,
                     5, 0, 0,
                     1, 1, 0]
    
    def get_tt(dr):
        dr = dr.removesuffix('-0.01').removesuffix('-0.1').removesuffix('-1.0')
        dr = float(dr)
        tt, _ = ans.convert_diffcoeff_to_transittimes(dr, 250)
        return f'\nsimulated trace\n$\\tau_{{sim}}={tt:.2f}ms$'
    
    def save_plot(filename, txt):
        plot_file = f'{filename}_{txt}'.replace(' ', '_').replace(
            '\n', '_').replace('"', '').replace('{', '').replace(
            '}', '').replace('$', '').replace('=', '-').replace('\\', '')
        plt.savefig(f'{plot_file}.pdf', bbox_inches='tight', dpi=300)
        os.system(f'pdf2svg {plot_file}.pdf {plot_file}.svg')
        os.system(f'rm {plot_file}.pdf')
    
    def plot_predictions(model_id, filename):
        def plot_cluster_prediction(filename):
            for i, (idx, t) in enumerate(zip(plot1_index, plot1_traceno)):
                fig = plt.figure()
                ax = plt.subplot(111)
                txt = get_tt(idx)
                ax.set_prop_cycle(color=[sns.color_palette()[3]])
                sns.lineplot(data=sim_pred.loc[:, idx].iloc[:, t],
                             label='prediction')
                plt.axhline(y=pred_thresh, xmin=0, xmax=1,
                            label='\nprediction\nthreshold',
                            color=sns.color_palette()[7], linestyle='--')
                plt.legend(bbox_to_anchor=(1.02, 1), loc='upper left', borderaxespad=0)
                plt.setp(ax, xlabel=r'Time [$ms$]', ylabel='artifact probability',
                         title=txt, ylim=[0, 1])
                save_plot(filename, f'prediction_{txt}_{i}')
    
        def plot_prediction_based_segmentation(filename):
            for i, (idx, t) in enumerate(zip(plot1_index, plot1_traceno)):
                fig = plt.figure()
                ax = plt.subplot(111)
                txt = get_tt(idx)
                ax.set_prop_cycle(color=[sns.color_palette()[3]])
                sim_predbool_scaled = sim_dirty.loc[:, idx].iloc[:, t].max() * sim_predbool.loc[:, idx].iloc[:, t]
                sns.lineplot(data=sim_predbool_scaled, alpha=0.5)
                plt.fill_between(x=sim_predbool.loc[:, idx].iloc[:, t].index,
                                 y1=sim_predbool_scaled,
                                 y2=0, alpha=0.5, label='prediction:\npeak artifacts')
    
                ax.set_prop_cycle(color=[sns.color_palette()[2]])
                sim_invpred_scaled = sim_dirty.loc[:, idx].iloc[:, t].max() * ~sim_predbool.loc[:, idx].iloc[:, t]
                plt.fill_between(x=sim_predbool.loc[:, idx].iloc[:, t].index,
                                 y1=sim_invpred_scaled,
                                 y2=0, alpha=0.5, label='\nprediction:\nno artifacts')
    
                ax.set_prop_cycle(color=[sns.color_palette()[0]])
                sns.lineplot(data=sim_dirty.loc[:, idx].iloc[:, t], label=txt)
                plt.legend(bbox_to_anchor=(1.02, 1), loc='upper left', borderaxespad=0)
                plt.setp(ax, xlabel=r'Time [$ms$]', ylabel=r'Intensity [a.u.]', title='')
                save_plot(filename, f'predseg_{txt}_{i}')
    
        def plot_prediction_based_cut_and_shift_correction(filename):
            for i, (idx, t) in enumerate(zip(plot1_index, plot1_traceno)):
                fig = plt.figure()
                ax = plt.subplot(111)
                txt = get_tt(idx)
                ax.set_prop_cycle(color=[sns.color_palette()[0]])
                sns.lineplot(data=sim_corr.loc[:, idx].iloc[:, t], label=txt)
                plt.legend(bbox_to_anchor=(1.02, 1), loc='upper left', borderaxespad=0)
                plt.setp(ax, xlabel=r'Time [$ms$]', ylabel=r'Intensity [a.u.]', title='')
                save_plot(filename, f'predcas_{txt}_{i}')
    
        logged_scaler = Path(f'/home/lex/Programme/drmed-git/data/mlruns/10/{model_ls[model_id]}/params/scaler')
        logged_scaler = !cat $logged_scaler
        logged_scaler = logged_scaler[0]
    
        logged_model = Path(f'/home/lex/Programme/drmed-git/data/mlruns/10/{model_ls[model_id]}/artifacts/model')
        logged_model = load_model(logged_model, compile=False)
        logged_model.compile(loss=bm.binary_ce_dice_loss(),
                             optimizer=Adam(),
                             metrics = bm.unet_metrics([0.1, 0.3, 0.5, 0.7, 0.9]))
    
        sim_dirty_prepro = ppd.convert_to_tfds_for_unet(sim_dirty)
        sim_dirty_prepro = ppd.scale_pad_and_batch_tfds_for_unet(
            sim_dirty_prepro, scaler=logged_scaler)
        sim_pred = logged_model.predict(sim_dirty_prepro, verbose=0)
        sim_pred = pd.DataFrame(sim_pred.squeeze(axis=2)).T
        sim_pred.columns = sim_columns
        sim_predbool = sim_pred > pred_thresh
    
        sim_corr = pd.DataFrame()
        for i in range(len(sim_dirty.columns)):
            sim_corr_trace = np.delete(sim_dirty.iloc[:, i].values,
                                       sim_predbool.iloc[:, i].values)
            sim_corr_trace = pd.DataFrame(sim_corr_trace)
            sim_corr = pd.concat([sim_corr, sim_corr_trace], axis='columns')
    
        sim_corr.columns = sim_columns
    
        plot_cluster_prediction(filename)
        plot_prediction_based_segmentation(filename)
        plot_prediction_based_cut_and_shift_correction(filename)
        plt.close('all')
    
    
  • first model: ff67be0b68e540a9a29a36a2d0c7a5be
    plot_predictions(model_id=0,
                     filename='data/exp-220227-unet/jupyter/analysis2_ff67b')
    
    • model predictions

      analysis2_ff67b_slow_molecules:_slow_cluster_prediction.svg analysis2_ff67b_slow_molecules:_medium_cluster_prediction.svg analysis2_ff67b_slow_molecules:_fast_cluster_prediction.svg analysis2_ff67b_medium_molecules:_slow_cluster_prediction.svg analysis2_ff67b_medium_molecules:_medium_cluster_prediction.svg analysis2_ff67b_medium_molecules:_fast_cluster_prediction.svg analysis2_ff67b_fast_molecules:_slow_cluster_prediction.svg analysis2_ff67b_fast_molecules:_medium_cluster_prediction.svg analysis2_ff67b_fast_molecules:_fast_cluster_prediction.svg

    • prediction-based segmentation (threshold=0.5)

      analysis2_ff67b_slow_molecules_and_slow_clusters:_prediction-based_segmentation.svg analysis2_ff67b_slow_molecules_and_medium_clusters:_prediction-based_segmentation.svg analysis2_ff67b_slow_molecules_and_fast_clusters:_prediction-based_segmentation.svg analysis2_ff67b_medium_molecules_and_slow_clusters:_prediction-based_segmentation.svg analysis2_ff67b_medium_molecules_and_medium_clusters:_prediction-based_segmentation.svg analysis2_ff67b_medium_molecules_and_fast_clusters:_prediction-based_segmentation.svg analysis2_ff67b_fast_molecules_and_slow_clusters:_prediction-based_segmentation.svg analysis2_ff67b_fast_molecules_and_medium_clusters:_prediction-based_segmentation.svg analysis2_ff67b_fast_molecules_and_fast_clusters:_prediction-based_segmentation.svg

    • prediction-based cut-and-stitch correction

      analysis2_ff67b_slow_molecules_and_slow_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_ff67b_slow_molecules_and_medium_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_ff67b_slow_molecules_and_fast_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_ff67b_medium_molecules_and_slow_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_ff67b_medium_molecules_and_medium_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_ff67b_medium_molecules_and_fast_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_ff67b_fast_molecules_and_slow_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_ff67b_fast_molecules_and_medium_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_ff67b_fast_molecules_and_fast_clusters:_prediction-based_cut_and_shift_correction.svg

  • second model: 347669d050f344ad9fb9e480c814f727
    plot_predictions(model_id=1,
                     filename='data/exp-220227-unet/jupyter/analysis2_34766')
    
    
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    /tmp/ipykernel_5495/3600776436.py:87: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
      fig = plt.figure()
    
    • model predictions

      analysis2_34766_slow_molecules:_slow_cluster_prediction.svg analysis2_34766_slow_molecules:_medium_cluster_prediction.svg analysis2_34766_slow_molecules:_fast_cluster_prediction.svg analysis2_34766_medium_molecules:_slow_cluster_prediction.svg analysis2_34766_medium_molecules:_medium_cluster_prediction.svg analysis2_34766_medium_molecules:_fast_cluster_prediction.svg analysis2_34766_fast_molecules:_slow_cluster_prediction.svg analysis2_34766_fast_molecules:_medium_cluster_prediction.svg analysis2_34766_fast_molecules:_fast_cluster_prediction.svg

    • prediction-based segmentation (threshold=0.5)

      analysis2_34766_slow_molecules_and_slow_clusters:_prediction-based_segmentation.svg analysis2_34766_slow_molecules_and_medium_clusters:_prediction-based_segmentation.svg analysis2_34766_slow_molecules_and_fast_clusters:_prediction-based_segmentation.svg analysis2_34766_medium_molecules_and_slow_clusters:_prediction-based_segmentation.svg analysis2_34766_medium_molecules_and_medium_clusters:_prediction-based_segmentation.svg analysis2_34766_medium_molecules_and_fast_clusters:_prediction-based_segmentation.svg analysis2_34766_fast_molecules_and_slow_clusters:_prediction-based_segmentation.svg analysis2_34766_fast_molecules_and_medium_clusters:_prediction-based_segmentation.svg analysis2_34766_fast_molecules_and_fast_clusters:_prediction-based_segmentation.svg

    • prediction-based cut-and-stitch correction

      analysis2_34766_slow_molecules_and_slow_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34766_slow_molecules_and_medium_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34766_slow_molecules_and_fast_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34766_medium_molecules_and_slow_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34766_medium_molecules_and_medium_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34766_medium_molecules_and_fast_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34766_fast_molecules_and_slow_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34766_fast_molecules_and_medium_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34766_fast_molecules_and_fast_clusters:_prediction-based_cut_and_shift_correction.svg

  • third model: 714af8cd12c1441eac4ca980e8c20070
    plot_predictions(model_id=2,
                     filename='data/exp-220227-unet/jupyter/analysis2_714af')
    
    fig = plt.figure()
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    /tmp/ipykernel_5495/3600776436.py:87: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
      fig = plt.figure()
    
    • model predictions

      analysis2_714af_slow_molecules:_slow_cluster_prediction.svg analysis2_714af_slow_molecules:_medium_cluster_prediction.svg analysis2_714af_slow_molecules:_fast_cluster_prediction.svg analysis2_714af_medium_molecules:_slow_cluster_prediction.svg analysis2_714af_medium_molecules:_medium_cluster_prediction.svg analysis2_714af_medium_molecules:_fast_cluster_prediction.svg analysis2_714af_fast_molecules:_slow_cluster_prediction.svg analysis2_714af_fast_molecules:_medium_cluster_prediction.svg analysis2_714af_fast_molecules:_fast_cluster_prediction.svg

    • prediction-based segmentation (threshold=0.5)

      analysis2_714af_slow_molecules_and_slow_clusters:_prediction-based_segmentation.svg analysis2_714af_slow_molecules_and_medium_clusters:_prediction-based_segmentation.svg analysis2_714af_slow_molecules_and_fast_clusters:_prediction-based_segmentation.svg analysis2_714af_medium_molecules_and_slow_clusters:_prediction-based_segmentation.svg analysis2_714af_medium_molecules_and_medium_clusters:_prediction-based_segmentation.svg analysis2_714af_medium_molecules_and_fast_clusters:_prediction-based_segmentation.svg analysis2_714af_fast_molecules_and_slow_clusters:_prediction-based_segmentation.svg analysis2_714af_fast_molecules_and_medium_clusters:_prediction-based_segmentation.svg analysis2_714af_fast_molecules_and_fast_clusters:_prediction-based_segmentation.svg

    • prediction-based cut-and-stitch correction

      analysis2_714af_slow_molecules_and_slow_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_714af_slow_molecules_and_medium_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_714af_slow_molecules_and_fast_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_714af_medium_molecules_and_slow_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_714af_medium_molecules_and_medium_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_714af_medium_molecules_and_fast_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_714af_fast_molecules_and_slow_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_714af_fast_molecules_and_medium_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_714af_fast_molecules_and_fast_clusters:_prediction-based_cut_and_shift_correction.svg

  • fourth model: 34a6d207ac594035b1009c330fb67a65
    plot_predictions(model_id=3,
                     filename='data/exp-220227-unet/jupyter/analysis2_34a6d')
    
    
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    /tmp/ipykernel_5495/3600776436.py:87: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
      fig = plt.figure()
    
    • model predictions

      analysis2_34a6d_slow_molecules:_slow_cluster_prediction.svg analysis2_34a6d_slow_molecules:_medium_cluster_prediction.svg analysis2_34a6d_slow_molecules:_fast_cluster_prediction.svg analysis2_34a6d_medium_molecules:_slow_cluster_prediction.svg analysis2_34a6d_medium_molecules:_medium_cluster_prediction.svg analysis2_34a6d_medium_molecules:_fast_cluster_prediction.svg analysis2_34a6d_fast_molecules:_slow_cluster_prediction.svg analysis2_34a6d_fast_molecules:_medium_cluster_prediction.svg analysis2_34a6d_fast_molecules:_fast_cluster_prediction.svg

    • prediction-based segmentation (threshold=0.5)

      analysis2_34a6d_slow_molecules_and_slow_clusters:_prediction-based_segmentation.svg analysis2_34a6d_slow_molecules_and_medium_clusters:_prediction-based_segmentation.svg analysis2_34a6d_slow_molecules_and_fast_clusters:_prediction-based_segmentation.svg analysis2_34a6d_medium_molecules_and_slow_clusters:_prediction-based_segmentation.svg analysis2_34a6d_medium_molecules_and_medium_clusters:_prediction-based_segmentation.svg analysis2_34a6d_medium_molecules_and_fast_clusters:_prediction-based_segmentation.svg analysis2_34a6d_fast_molecules_and_slow_clusters:_prediction-based_segmentation.svg analysis2_34a6d_fast_molecules_and_medium_clusters:_prediction-based_segmentation.svg analysis2_34a6d_fast_molecules_and_fast_clusters:_prediction-based_segmentation.svg

    • prediction-based cut-and-stitch correction

      analysis2_34a6d_slow_molecules_and_slow_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34a6d_slow_molecules_and_medium_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34a6d_slow_molecules_and_fast_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34a6d_medium_molecules_and_slow_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34a6d_medium_molecules_and_medium_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34a6d_medium_molecules_and_fast_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34a6d_fast_molecules_and_slow_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34a6d_fast_molecules_and_medium_clusters:_prediction-based_cut_and_shift_correction.svg analysis2_34a6d_fast_molecules_and_fast_clusters:_prediction-based_cut_and_shift_correction.svg

  • fifth model: 0cd2023eeaf745aca0d3e8ad5e1fc653. I re-wrote the plotting script a bit.
    plot_predictions(model_id=5,
                     filename='data/exp-220227-unet/jupyter/analysis2_0cd20')
    
    
    2022-08-13 17:00:55.175177: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2022-08-13 17:00:55.175214: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
    2022-08-13 17:00:55.175247: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (Topialex): /proc/driver/nvidia/version does not exist
    2022-08-13 17:00:55.177043: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    2022-08-13 17:01:06.559383: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 98304000 exceeds 10% of free system memory.
    2022-08-13 17:01:06.640827: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 98304000 exceeds 10% of free system memory.
    2022-08-13 17:01:07.439633: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 98304000 exceeds 10% of free system memory.
    2022-08-13 17:01:07.700351: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 98304000 exceeds 10% of free system memory.2022-08-13 17:03:32.777028: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 98304000 exceeds 10% of free system memory.
    /tmp/ipykernel_6898/3697032523.py:92: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
      fig = plt.figure()
    
    • model predictions

      analysis2_0cd20_prediction__simulated_trace_tau_sim-3.76ms_0.svg analysis2_0cd20_prediction__simulated_trace_tau_sim-3.76ms_1.svg analysis2_0cd20_prediction__simulated_trace_tau_sim-3.76ms_2.svg analysis2_0cd20_prediction__simulated_trace_tau_sim-56.36ms_3.svg analysis2_0cd20_prediction__simulated_trace_tau_sim-56.36ms_4.svg analysis2_0cd20_prediction__simulated_trace_tau_sim-56.36ms_5.svg analysis2_0cd20_prediction__simulated_trace_tau_sim-163.35ms_6.svg analysis2_0cd20_prediction__simulated_trace_tau_sim-163.35ms_7.svg analysis2_0cd20_prediction__simulated_trace_tau_sim-163.35ms_8.svg

    • prediction-based segmentation (threshold=0.5)

      analysis2_0cd20_predseg__simulated_trace_tau_sim-3.76ms_0.svg analysis2_0cd20_predseg__simulated_trace_tau_sim-3.76ms_1.svg analysis2_0cd20_predseg__simulated_trace_tau_sim-3.76ms_2.svg analysis2_0cd20_predseg__simulated_trace_tau_sim-56.36ms_3.svg analysis2_0cd20_predseg__simulated_trace_tau_sim-56.36ms_4.svg analysis2_0cd20_predseg__simulated_trace_tau_sim-56.36ms_5.svg analysis2_0cd20_predseg__simulated_trace_tau_sim-163.35ms_6.svg analysis2_0cd20_predseg__simulated_trace_tau_sim-163.35ms_7.svg analysis2_0cd20_predseg__simulated_trace_tau_sim-163.35ms_8.svg

    • prediction-based cut-and-stitch correction

      analysis2_0cd20_predcas__simulated_trace_tau_sim-3.76ms_0.svg analysis2_0cd20_predcas__simulated_trace_tau_sim-3.76ms_1.svg analysis2_0cd20_predcas__simulated_trace_tau_sim-3.76ms_2.svg analysis2_0cd20_predcas__simulated_trace_tau_sim-56.36ms_3.svg analysis2_0cd20_predcas__simulated_trace_tau_sim-56.36ms_4.svg analysis2_0cd20_predcas__simulated_trace_tau_sim-56.36ms_5.svg analysis2_0cd20_predcas__simulated_trace_tau_sim-163.35ms_6.svg analysis2_0cd20_predcas__simulated_trace_tau_sim-163.35ms_7.svg analysis2_0cd20_predcas__simulated_trace_tau_sim-163.35ms_8.svg

  • fifth model: fe81d71c52404ed790b3a32051258da9
    plot_predictions(model_id=6,
                     filename='data/exp-220227-unet/jupyter/analysis2_fe81d')
    
    
    2023-02-16 13:42:36.880181: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2023-02-16 13:42:36.882860: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
    2023-02-16 13:42:36.885399: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (Topialex): /proc/driver/nvidia/version does not exist
    2023-02-16 13:42:36.911523: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    2023-02-16 13:42:48.544143: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 98304000 exceeds 10% of free system memory.
    2023-02-16 13:42:48.656459: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 98304000 exceeds 10% of free system memory.
    2023-02-16 13:42:49.489339: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 98304000 exceeds 10% of free system memory.
    2023-02-16 13:42:50.028156: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 98304000 exceeds 10% of free system memory.2023-02-16 13:52:50.335793: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 98304000 exceeds 10% of free system memory.
    /tmp/ipykernel_14923/851958287.py:66: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
      fig = plt.figure()
    
    • model predictions

      analysis2_fe81d_prediction__simulated_trace_tau_sim-3.76ms_0.svg analysis2_fe81d_prediction__simulated_trace_tau_sim-3.76ms_1.svg analysis2_fe81d_prediction__simulated_trace_tau_sim-3.76ms_2.svg analysis2_fe81d_prediction__simulated_trace_tau_sim-56.36ms_3.svg analysis2_fe81d_prediction__simulated_trace_tau_sim-56.36ms_4.svg analysis2_fe81d_prediction__simulated_trace_tau_sim-56.36ms_5.svg analysis2_fe81d_prediction__simulated_trace_tau_sim-163.35ms_6.svg analysis2_fe81d_prediction__simulated_trace_tau_sim-163.35ms_7.svg analysis2_fe81d_prediction__simulated_trace_tau_sim-163.35ms_8.svg

    • prediction-based segmentation (threshold=0.5)

      analysis2_fe81d_predseg__simulated_trace_tau_sim-3.76ms_0.svg analysis2_fe81d_predseg__simulated_trace_tau_sim-3.76ms_1.svg analysis2_fe81d_predseg__simulated_trace_tau_sim-3.76ms_2.svg analysis2_fe81d_predseg__simulated_trace_tau_sim-56.36ms_3.svg analysis2_fe81d_predseg__simulated_trace_tau_sim-56.36ms_4.svg analysis2_fe81d_predseg__simulated_trace_tau_sim-56.36ms_5.svg analysis2_fe81d_predseg__simulated_trace_tau_sim-163.35ms_6.svg analysis2_fe81d_predseg__simulated_trace_tau_sim-163.35ms_7.svg analysis2_fe81d_predseg__simulated_trace_tau_sim-163.35ms_8.svg

    • prediction-based cut-and-stitch correction

      analysis2_fe81d_predcas__simulated_trace_tau_sim-3.76ms_0.svg analysis2_fe81d_predcas__simulated_trace_tau_sim-3.76ms_1.svg analysis2_fe81d_predcas__simulated_trace_tau_sim-3.76ms_2.svg analysis2_fe81d_predcas__simulated_trace_tau_sim-56.36ms_3.svg analysis2_fe81d_predcas__simulated_trace_tau_sim-56.36ms_4.svg analysis2_fe81d_predcas__simulated_trace_tau_sim-56.36ms_5.svg analysis2_fe81d_predcas__simulated_trace_tau_sim-163.35ms_6.svg analysis2_fe81d_predcas__simulated_trace_tau_sim-163.35ms_7.svg analysis2_fe81d_predcas__simulated_trace_tau_sim-163.35ms_8.svg

  • I did not do these plots for models 19e3e or c1204, because the did not work well in latex table (See below)
2.6.11.2 latex table from sim results
  • All correlation curve fitting from Analysis 1b and 1c was done in a structured manner via focuspoint.FCS_point_correlator from Dominic Waithe.
  • Now we load the modules and tell the code where to find the fit results
  • these fit results are the average of 300 curves per simulated group of fast molecule diffusion coefficients \(\{0.069, 0.08, 0.1, 0.2, 0.4, 0.6, 1.0, 3.0, 10, 50\}\mu m^2/s\)
  • this is only an illustration, because the actual distribution of each of the 300 fitted curves is the interesting result. This is shown in exp-220316-publication1
  • here, the latex export is the cool thing - and the comparison of all models!
    %cd ~/Programme/drmed-git
    
    /home/lex/Programme/drmed-git
    
    import os
    import sys
    import numpy as np
    import matplotlib.pyplot as plt
    import pandas as pd
    import seaborn as sns
    from pathlib import Path
    from pprint import pprint
    
    FLUOTRACIFY_PATH = '/home/lex/Programme/drmed-git/src/'
    sys.path.append(FLUOTRACIFY_PATH)
    from fluotracify.simulations import (
       analyze_simulations as ans,
    )
    
    # use seaborn style as default even if I just use matplotlib
    
    sns.set_theme(style="whitegrid", font_scale=2, palette='colorblind',
                  context='paper')
    # logging.basicConfig(filename="/home/lex/Programme/drmed-git/data/exp-220227-unet/jupyter.log",
    #               filemode='w', format='%(asctime)s - %(message)s',
    #               force=True,
    #               level=logging.DEBUG)
    
    model_ls = ['ff67be0b68e540a9a29a36a2d0c7a5be', '347669d050f344ad9fb9e480c814f727',
                '714af8cd12c1441eac4ca980e8c20070', '34a6d207ac594035b1009c330fb67a65',
                '484af471c61943fa90e5f78e78a229f0', '0cd2023eeaf745aca0d3e8ad5e1fc653',
                'fe81d71c52404ed790b3a32051258da9', '19e3e786e1bc4e2b93856f5dc9de8216',
                'c1204e3a8a1e4c40a35b5b7b1922d1ce']
    
    model_name_ls = [f'{s:.5}' for s in model_ls]
    
    2023-02-15 22:29:36.789486: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2023-02-15 22:29:36.789635: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    
    path1 = Path('data/exp-220227-unet/2022-04-25_simulations')
    path2 = Path('data/exp-220316-publication1/220517_simulations')
    
    odot069_models1_path = path1 / '0.069-all-results/0dot069_all_1comp_outputParam.csv'
    odot069_models2_path = path1 / '0.069-all-results/0dot069_all_2comp_outputParam.csv'
    odot069_corr1_path = path2 / '0.069-all-results/0dot069_lab-all_1comp_outputParam.csv'
    odot069_corr2_path = path2 / '0.069-all-results/0dot069_lab-all_2comp_outputParam.csv'
    
    odot08_models1_path = path1 / '0.08-all-results/0dot08_all_1comp_outputParam.csv'
    odot08_models2_path = path1 / '0.08-all-results/0dot08_all_2comp_outputParam.csv'
    odot08_corr1_path = path2 / '0.08-all-results/0dot08_lab-all_1comp_outputParam.csv'
    odot08_corr2_path = path2 / '0.08-all-results/0dot08_lab-all_2comp_outputParam.csv'
    
    odot1_models1_path = path1 / '0.1-all-results/0dot1_all_1comp_outputParam.csv'
    odot1_models2_path = path1 / '0.1-all-results/0dot1_all_2comp_outputParam.csv'
    odot1_corr1_path = path2 / '0.1-all-results/0dot1_lab-all_1comp_outputParam.csv'
    odot1_corr2_path = path2 / '0.1-all-results/0dot1_lab-all_2comp_outputParam.csv'
    
    odot2_models1_path = path1 / '0.2-all-results/0dot2_all_1comp_outputParam.csv'
    odot2_models2_path = path1 / '0.2-all-results/0dot2_all_2comp_outputParam.csv'
    odot2_corr1_path = path2 / '0.2-all-results/0dot2_lab-all_1comp_outputParam.csv'
    odot2_corr2_path = path2 / '0.2-all-results/0dot2_lab-all_2comp_outputParam.csv'
    
    odot4_models1_path = path1 / '0.4-all-results/0dot4_all_1comp_outputParam.csv'
    odot4_models2_path = path1 / '0.4-all-results/0dot4_all_2comp_outputParam.csv'
    odot4_corr1_path = path2 / '0.4-all-results/0dot4_lab-all_1comp_outputParam.csv'
    odot4_corr2_path = path2 / '0.4-all-results/0dot4_lab-all_2comp_outputParam.csv'
    
    odot6_models1_path = path1 / '0.6-all-results/0dot6_all_1comp_outputParam.csv'
    odot6_models2_path = path1 / '0.6-all-results/0dot6_all_2comp_outputParam.csv'
    odot6_corr1_path = path2 / '0.6-all-results/0dot6_lab-all_1comp_outputParam.csv'
    odot6_corr2_path = path2 / '0.6-all-results/0dot6_lab-all_2comp_outputParam.csv'
    
    one_models1_path = path1 / '1.0-all-results/1dot0_all_1comp_outputParam.csv'
    one_models2_path = path1 / '1.0-all-results/1dot0_all_2comp_outputParam.csv'
    one_corr1_path = path2 / '1.0-all-results/1dot0_lab-all_1comp_outputParam.csv'
    one_corr2_path = path2 / '1.0-all-results/1dot0_lab-all_2comp_outputParam.csv'
    
    three_models1_path = path1 / '3.0-all-results/3dot0_all_1comp_outputParam.csv'
    three_models2_path = path1 / '3.0-all-results/3dot0_all_2comp_outputParam.csv'
    three_corr1_path = path2 / '3.0-all-results/3dot0_lab-all_1comp_outputParam.csv'
    three_corr2_path = path2 / '3.0-all-results/3dot0_lab-all_2comp_outputParam.csv'
    
    ten_models1_path = path1 / '10.0-all-results/10dot0_all_1comp_outputParam.csv'
    ten_models2_path = path1 / '10.0-all-results/10dot0_all_2comp_outputParam.csv'
    ten_corr1_path = path2 / '10.0-all-results/10dot0_lab-all_1comp_outputParam.csv'
    ten_corr2_path = path2 / '10.0-all-results/10dot0_lab-all_2comp_outputParam.csv'
    
    fifty_models1_path = path1 / '50.0-all-results/50dot0_all_1comp_outputParam.csv'
    fifty_models2_path = path1 / '50.0-all-results/50dot0_all_2comp_outputParam.csv'
    fifty_corr1_path = path2 / '50.0-all-results/50dot0_lab-all_1comp_outputParam.csv'
    fifty_corr2_path = path2 / '50.0-all-results/50dot0_lab-all_2comp_outputParam.csv'
    
    
  • we load the data and combine it. We convert the simulated D [um2/s] in t[ms]. Based on an exploratory plot of A1 and A2 we swap A1-A2 and txy1-txy2, so that A1 and txy1 always represent the bigger fitted fraction size.
  • I wrote a latex table export. The table should display cells with a green background, when the correction method is inside an accepted range of +-log 10%, else it should have an orange background.
  • I prepared tables for two occasions:
    1. a short table for a beamer presentation
    2. a long table for a report
    tt_txt = 0.069
    odot069_models1 = pd.read_csv(odot069_models1_path, sep=',').assign(
        components=10*[1,], sim=10*[tt_txt])
    odot069_models2 = pd.read_csv(odot069_models2_path, sep=',').assign(
        components=10*[2,], sim=10*[tt_txt])
    odot069_corr1 = pd.read_csv(odot069_corr1_path, sep=',').assign(
        components=2*[1,], sim=2*[tt_txt])
    odot069_corr2 = pd.read_csv(odot069_corr2_path, sep=',').assign(
        components=2*[2,], sim=2*[tt_txt])
    
    tt_txt = 0.08
    odot08_models1 = pd.read_csv(odot08_models1_path, sep=',').assign(
        components=10*[1,], sim=10*[tt_txt])
    odot08_models2 = pd.read_csv(odot08_models2_path, sep=',').assign(
        components=10*[2,], sim=10*[tt_txt])
    odot08_corr1 = pd.read_csv(odot08_corr1_path, sep=',').assign(
        components=2*[1,], sim=2*[tt_txt])
    odot08_corr2 = pd.read_csv(odot08_corr2_path, sep=',').assign(
        components=2*[2,], sim=2*[tt_txt])
    
    tt_txt = 0.1
    odot1_models1 = pd.read_csv(odot1_models1_path, sep=',').assign(
        components=10*[1,], sim=10*[tt_txt])
    odot1_models2 = pd.read_csv(odot1_models2_path, sep=',').assign(
        components=10*[2,], sim=10*[tt_txt])
    odot1_corr1 = pd.read_csv(odot1_corr1_path, sep=',').assign(
        components=2*[1,], sim=2*[tt_txt])
    odot1_corr2 = pd.read_csv(odot1_corr2_path, sep=',').assign(
        components=2*[2,], sim=2*[tt_txt])
    
    tt_txt = 0.2
    odot2_models1 = pd.read_csv(odot2_models1_path, sep=',').assign(
        components=10*[1,], sim=10*[tt_txt])
    odot2_models2 = pd.read_csv(odot2_models2_path, sep=',').assign(
        components=10*[2,], sim=10*[tt_txt])
    odot2_corr1 = pd.read_csv(odot2_corr1_path, sep=',').assign(
        components=2*[1,], sim=2*[tt_txt])
    odot2_corr2 = pd.read_csv(odot2_corr2_path, sep=',').assign(
        components=2*[2,], sim=2*[tt_txt])
    
    tt_txt = 0.4
    odot4_models1 = pd.read_csv(odot4_models1_path, sep=',').assign(
        components=10*[1,], sim=10*[tt_txt])
    odot4_models2 = pd.read_csv(odot4_models2_path, sep=',').assign(
        components=10*[2,], sim=10*[tt_txt])
    odot4_corr1 = pd.read_csv(odot4_corr1_path, sep=',').assign(
        components=2*[1,], sim=2*[tt_txt])
    odot4_corr2 = pd.read_csv(odot4_corr2_path, sep=',').assign(
        components=2*[2,], sim=2*[tt_txt])
    
    tt_txt = 0.6
    odot6_models1 = pd.read_csv(odot6_models1_path, sep=',').assign(
        components=10*[1,], sim=10*[tt_txt])
    odot6_models2 = pd.read_csv(odot6_models2_path, sep=',').assign(
        components=10*[2,], sim=10*[tt_txt])
    odot6_corr1 = pd.read_csv(odot6_corr1_path, sep=',').assign(
        components=2*[1,], sim=2*[tt_txt])
    odot6_corr2 = pd.read_csv(odot6_corr2_path, sep=',').assign(
        components=2*[2,], sim=2*[tt_txt])
    
    tt_txt = 1
    one_models1 = pd.read_csv(one_models1_path, sep=',').assign(
        components=10*[1,], sim=10*[tt_txt])
    one_models2 = pd.read_csv(one_models2_path, sep=',').assign(
        components=10*[2,], sim=10*[tt_txt])
    one_corr1 = pd.read_csv(one_corr1_path, sep=',').assign(
        components=2*[1,], sim=2*[tt_txt])
    one_corr2 = pd.read_csv(one_corr2_path, sep=',').assign(
        components=2*[2,], sim=2*[tt_txt])
    
    tt_txt = 3
    three_models1 = pd.read_csv(three_models1_path, sep=',').assign(
        components=10*[1,], sim=10*[tt_txt])
    three_models2 = pd.read_csv(three_models2_path, sep=',').assign(
        components=10*[2,], sim=10*[tt_txt])
    three_corr1 = pd.read_csv(three_corr1_path, sep=',').assign(
        components=2*[1,], sim=2*[tt_txt])
    three_corr2 = pd.read_csv(three_corr2_path, sep=',').assign(
        components=2*[2,], sim=2*[tt_txt])
    
    tt_txt = 10
    ten_models1 = pd.read_csv(ten_models1_path, sep=',').assign(
        components=10*[1,], sim=10*[tt_txt])
    ten_models2 = pd.read_csv(ten_models2_path, sep=',').assign(
        components=10*[2,], sim=10*[tt_txt])
    ten_corr1 = pd.read_csv(ten_corr1_path, sep=',').assign(
        components=2*[1,], sim=2*[tt_txt])
    ten_corr2 = pd.read_csv(ten_corr2_path, sep=',').assign(
        components=2*[2,], sim=2*[tt_txt])
    
    tt_txt = 50
    # fifty_models1 = pd.read_csv(fifty_models1_path, sep=',',
    #     index_col='name_of_plot').assign(components=10*[1,])
    # fifty_models2 = pd.read_csv(fifty_models2_path, sep=',',
    #     index_col='name_of_plot').assign(components=10*[2,])
    fifty_corr1 = pd.read_csv(fifty_corr1_path, sep=',').assign(
        components=2*[1,], sim=2*[tt_txt])
    fifty_corr2 = pd.read_csv(fifty_corr2_path, sep=',').assign(
        components=2*[2,], sim=2*[tt_txt])
    
    all_param = pd.concat([odot069_models1, odot069_models2,
                           odot069_corr1, odot069_corr2,
                           odot08_models1, odot08_models2,
                           odot08_corr1, odot08_corr2,
                           odot1_models2, odot1_models1,
                           odot1_corr2, odot1_corr1,
                           odot2_models1, odot2_models2,
                           odot2_corr1, odot2_corr2,
                           odot4_models1, odot4_models2,
                           odot4_corr1, odot4_corr2,
                           odot6_models1, odot6_models2,
                           odot6_corr1, odot6_corr2,
                           one_models1, one_models2,
                           one_corr1, one_corr2,
                           three_models1, three_models2,
                           three_corr1, three_corr2])
                           # ten_models1, ten_models2,
                           # ten_corr1, ten_corr2,
                           # fifty_models1, fifty_models2,
                           # fifty_corr1, fifty_corr2
    
    
    def diffcoeff_to_transittimes(diff, prec):
        tt, tt_low_high = ans.convert_diffcoeff_to_transittimes(diff, fwhm=250)
        # tt_txt = f'\\makecell{{${tt:.{prec}f}$\\\\$\\numrange{{{tt_low_high[0]:.{prec}f}}}{{{tt_low_high[1]:.{prec}f}}}$}}' # \\\\$(\\sim {diff})$
        tt_txt = f'\\makecell{{${tt:.{prec}f}$\\\\$\\textbf{{{tt_low_high[0]:.{prec}f}-{tt_low_high[1]:.{prec}f}}}$}}' # \\\\$(\\sim {diff})$
        return tt_txt
    
    def sort_fit(param_ls):
        sim = param_ls[-1]
        _, tt_low_high = ans.convert_diffcoeff_to_transittimes(sim, fwhm=250)
        array = np.array(list(param_ls)[:-1]).reshape((2, 2))
        # sort by transit times
        array = array[:, array[0, :].argsort()]
        A_fast = array[1, 0]
        A_slow = array[1, 1]
        t_fast = array[0, 0]
        t_slow = array[0, 1]
        if np.isnan(t_slow):
            if tt_low_high[0] <= t_fast <= tt_low_high[1]:
                # out = f'\\cellcolor[HTML]{{009e73}}${t_fast:.2f}$'
                # for beamer:
                out = f'\\colorbox[HTML]{{009e73}}{{\\makecell{{${t_fast:.2f}$}}}}'
            else:
                # out = f'\\cellcolor[HTML]{{d55e00}}${t_fast:.2f}$'
                out = f'\\colorbox[HTML]{{d55e00}}{{\\makecell{{${t_fast:.2f}$}}}}'
    
        elif f'{A_fast:.0%}' == '100%':
            if tt_low_high[0] <= t_fast <= tt_low_high[1]:
                # out = f'\\cellcolor[HTML]{{009e73}}${t_fast:.2f}(100\\%)$'
                out = f'\\colorbox[HTML]{{009e73}}{{\\makecell{{${t_fast:.2f}$}}}}'
            else:
                # out = f'\\cellcolor[HTML]{{d55e00}}${t_fast:.2f}(100\\%)$'
                out = f'\\colorbox[HTML]{{d55e00}}{{\\makecell{{${t_fast:.2f}$}}}}'
        elif f'{A_slow:.0%}' == '100%':
            if tt_low_high[0] <= t_slow <= tt_low_high[1]:
                # out = f'\\cellcolor[HTML]{{009e73}}${t_slow:.2f}(100\\%)$'
                out = f'\\colorbox[HTML]{{009e73}}{{\\makecell{{${t_slow:.2f}$}}}}'
            else:
                # out = f'\\cellcolor[HTML]{{d55e00}}${t_slow:.2f}(100\\%)$'
                out = f'\\colorbox[HTML]{{d55e00}}{{\\makecell{{${t_slow:.2f}$}}}}'
        else:
            if (tt_low_high[0] <= t_fast <= tt_low_high[1]) or (
                tt_low_high[0] <= t_slow <= tt_low_high[1]):
                # out = f'\\cellcolor[HTML]{{009e73}}\\colorbox[HTML]{{009e73}}{{\\makecell{{${t_fast:.2f}^f({A_fast:.0%})$\\\\'\
                #       f'${t_slow:.2f}^s({A_slow:.0%})$}}}}'
                out = f'\\colorbox[HTML]{{009e73}}{{\\makecell{{${t_fast:.2f}$\\\\'\
                      f'${t_slow:.2f}$}}}}'
    
            else:
                # out = f'\\cellcolor[HTML]{{d55e00}}\\colorbox[HTML]{{d55e00}}{{\\makecell{{${t_fast:.2f}^f({A_fast:.0%})$\\\\'\
                #       f'${t_slow:.2f}^s({A_slow:.0%})$}}}}'
                out = f'\\colorbox[HTML]{{d55e00}}{{\\makecell{{${t_fast:.2f}$\\\\'\
                      f'${t_slow:.2f}$}}}}'
    
            out = out.replace('%', '\\%')
        return out
    
    
    all_param['fit results'] = all_param[['txy1', 'txy2', 'A1', 'A2', 'sim']].apply(lambda x: sort_fit(x), axis=1)
    all_param = all_param[['name_of_plot', 'sim', 'components', 'fit results']]
    all_param = all_param.pivot_table(values='fit results',
                                      columns='sim',
                                      index=['name_of_plot', 'components'],
                                      aggfunc=lambda x: '-'.join(x))
    # for beamer: only dirty, delete, and cutandshift
    first = ['dirty', 'cutandshift', 'delete']
    # dirty, delete, cutandshift, and all models
    # first = ['dirty', 'cutandshift', 'delete'] + model_name_ls.copy()
    
    second = [1, 2]
    index_order = pd.MultiIndex.from_product([first, second],
                                             names=[r'\makecell{type of\\processing}', 'fit'])
    column_order = [0.069, 0.08, 0.1, 0.2, 0.4, 0.6, 1, 3] #, 10, 50]
    all_param = all_param.reindex(index=index_order, columns=column_order)
    all_param = all_param.rename_axis(columns={'sim' : r'\makecell{simulated \tau_D [\unit{\ms}]\\\log 10\% \text{ tol.}}'}) # \\(\sim D [\unit[per-mode = fraction]{\micro\metre\squared\per\second}])
                                      # index={'fit' : 'fit^*'})
    all_param = all_param.rename(index={'dirty' : r'\makecell{control:\\no correction}',
                                        'delete' : r'\makecell{old method:\\weight=0}',
                                        'cutandshift' : r'\makecell{new method:\\cut and shift}',
                                        'ff67b' : r'\makecell{prediction model:\\ff67b (\qty{14}{\mega\byte})}',
                                        '34766' : r'\makecell{prediction model:\\34766 (\qty{73}{\mega\byte})}',
                                        '714af' : r'\makecell{prediction model:\\714af (\qty{234}{\mega\byte})}',
                                        '34a6d' : r'\makecell{prediction model:\\34a6d (\qty{7}{\mega\byte})}',
                                        '484af' : r'\makecell{prediction model:\\484af (\qty{275}{\mega\byte})}',
                                        '0cd20' : r'\makecell{prediction model:\\0cd20 (\qty{200}{\mega\byte})}',
                                        'fe81d' : r'\makecell{prediction model:\\fe81d (\qty{186}{\mega\byte})}',
                                        '19e3e' : r'\makecell{prediction model:\\19e3e (\qty{172}{\mega\byte})}',
                                        'c1204' : r'\makecell{prediction model:\\c1204 (\qty{312}{\mega\byte})}',
                                        1 : r'\makecell{1 species}',
                                        2 : r'\makecell{2 - fast sp.\\2 - slow sp.}'},
                                  columns={0.069 : diffcoeff_to_transittimes(0.069, prec=0),
                                           0.08 : diffcoeff_to_transittimes(0.08, prec=0),
                                           0.1 : diffcoeff_to_transittimes(0.1, prec=0),
                                           0.2 : diffcoeff_to_transittimes(0.2, prec=0),
                                           0.4 : diffcoeff_to_transittimes(0.4, prec=0),
                                           0.6 : diffcoeff_to_transittimes(0.6, prec=0),
                                           1 : diffcoeff_to_transittimes(1, prec=1),
                                           3 : diffcoeff_to_transittimes(3, prec=2)})
                                           # 10 : diffcoeff_to_transittimes(10, prec=3),
                                           # 50 : diffcoeff_to_transittimes(50, prec=4)
    
    
    
    # for creation of multiindex:
    # predictions_mi = len(predictions) * ['prediction',]
    # column_multiindex = ['no correction', 'control', 'control'] + predictions_mi
    # column_tuple = list(zip(column_multiindex, column_order))
    # all_param.columns = pd.MultiIndex.from_tuples(column_tuple)
    
    # for nanoletters
    # with pd.option_context("max_colwidth", 1000):
    #     print(all_param.to_latex(escape=False,
    #                              column_format='cccccccccccccccccccc',
    #                              multicolumn_format='c',
    #                              longtable=True,
    #                              caption=('Justification of "cut and shift" in simulated fluorescence traces, and comparison of different models for prediction followed by "cut and shift" correction', 'Simulated data - correction results')))
    # for beamer
    with pd.option_context("max_colwidth", 1000):
        print(all_param.to_latex(escape=False,
                                 column_format='ccrrrrrrrrrrrrrrrrrr',
                                 multicolumn_format='c',
                                 caption=('"cut and shift" vs old correction method in simulated fluorescence traces', 'Simulated data - correction results')))
    
    
    all_param
    
    \begin{table}
    \centering
    \caption[Simulated data - correction results]{"cut and shift" vs old correction method in simulated fluorescence traces}
    \begin{tabular}{ccrrrrrrrrrrrrrrrrrr}
    \toprule
                                     & \makecell{simulated \tau_D [\unit{\ms}]\\\log 10\% \text{ tol.}} &                     \makecell{$163$\\$\textbf{98-272}$} &                     \makecell{$141$\\$\textbf{86-231}$} &                     \makecell{$113$\\$\textbf{70-181}$} &                       \makecell{$56$\\$\textbf{38-84}$} &                      \makecell{$28$\\$\textbf{20-39}$} &                      \makecell{$19$\\$\textbf{14-25}$} &                  \makecell{$11.3$\\$\textbf{8.8-14.4}$} &                \makecell{$3.76$\\$\textbf{3.29-4.29}$} \\
    \makecell{type of\\processing} & fit &                                                         &                                                         &                                                         &                                                         &                                                        &                                                        &                                                         &                                                        \\
    \midrule
    \makecell{control:\\no correction} & \makecell{1 species} &             \colorbox[HTML]{d55e00}{\makecell{$58.92$}} &            \colorbox[HTML]{009e73}{\makecell{$142.83$}} &             \colorbox[HTML]{009e73}{\makecell{$80.93$}} &            \colorbox[HTML]{d55e00}{\makecell{$145.00$}} &            \colorbox[HTML]{009e73}{\makecell{$27.55$}} &            \colorbox[HTML]{d55e00}{\makecell{$54.11$}} &            \colorbox[HTML]{d55e00}{\makecell{$110.39$}} &           \colorbox[HTML]{d55e00}{\makecell{$287.87$}} \\
                                     & \makecell{2 - fast sp.\\2 - slow sp.} &   \colorbox[HTML]{009e73}{\makecell{$15.85$\\$229.90$}} &   \colorbox[HTML]{d55e00}{\makecell{$70.02$\\$445.84$}} &   \colorbox[HTML]{009e73}{\makecell{$12.82$\\$138.39$}} &   \colorbox[HTML]{009e73}{\makecell{$53.66$\\$446.38$}} &  \colorbox[HTML]{d55e00}{\makecell{$13.17$\\$383.01$}} &  \colorbox[HTML]{009e73}{\makecell{$16.68$\\$424.67$}} &   \colorbox[HTML]{d55e00}{\makecell{$47.23$\\$284.90$}} &  \colorbox[HTML]{d55e00}{\makecell{$16.59$\\$643.73$}} \\
    \makecell{new method:\\cut and shift} & \makecell{1 species} &            \colorbox[HTML]{009e73}{\makecell{$161.43$}} &            \colorbox[HTML]{009e73}{\makecell{$130.03$}} &            \colorbox[HTML]{009e73}{\makecell{$100.67$}} &             \colorbox[HTML]{009e73}{\makecell{$53.11$}} &            \colorbox[HTML]{009e73}{\makecell{$26.25$}} &            \colorbox[HTML]{009e73}{\makecell{$17.43$}} &             \colorbox[HTML]{009e73}{\makecell{$11.52$}} &             \colorbox[HTML]{009e73}{\makecell{$3.62$}} \\
    & \makecell{2 - fast sp.\\2 - slow sp.} &  \colorbox[HTML]{009e73}{\makecell{$120.86$\\$429.73$}} &    \colorbox[HTML]{009e73}{\makecell{$4.33$\\$131.89$}} &  \colorbox[HTML]{009e73}{\makecell{$100.67$\\$100.68$}} &     \colorbox[HTML]{009e73}{\makecell{$0.00$\\$53.11$}} &    \colorbox[HTML]{009e73}{\makecell{$0.01$\\$26.34$}} &            \colorbox[HTML]{009e73}{\makecell{$17.43$}} &             \colorbox[HTML]{009e73}{\makecell{$11.52$}} &     \colorbox[HTML]{009e73}{\makecell{$3.62$\\$3.62$}} \\
    \makecell{old method:\\weight=0} & \makecell{1 species} &            \colorbox[HTML]{d55e00}{\makecell{$383.41$}} &            \colorbox[HTML]{d55e00}{\makecell{$253.55$}} &            \colorbox[HTML]{d55e00}{\makecell{$293.12$}} &            \colorbox[HTML]{d55e00}{\makecell{$301.92$}} &           \colorbox[HTML]{d55e00}{\makecell{$287.00$}} &           \colorbox[HTML]{d55e00}{\makecell{$143.32$}} &            \colorbox[HTML]{d55e00}{\makecell{$142.88$}} &           \colorbox[HTML]{d55e00}{\makecell{$362.80$}} \\
                                     & \makecell{2 - fast sp.\\2 - slow sp.} &   \colorbox[HTML]{d55e00}{\makecell{$28.66$\\$819.05$}} &  \colorbox[HTML]{d55e00}{\makecell{$45.97$\\$1158.57$}} &   \colorbox[HTML]{d55e00}{\makecell{$36.14$\\$723.50$}} &  \colorbox[HTML]{009e73}{\makecell{$44.09$\\$1733.81$}} &  \colorbox[HTML]{d55e00}{\makecell{$19.30$\\$747.77$}} &  \colorbox[HTML]{d55e00}{\makecell{$10.94$\\$211.09$}} &  \colorbox[HTML]{d55e00}{\makecell{$34.31$\\$1041.96$}} &  \colorbox[HTML]{d55e00}{\makecell{$20.37$\\$775.82$}} \\
    \bottomrule
    \end{tabular}
    \end{table}
    
      \makecell{simulated τD [\unit{\ms}]\\log 10\% \text{ tol.}} \makecell{$163$\\\(\textbf{98-272}\)} \makecell{$141$\\\(\textbf{86-231}\)} \makecell{$113$\\\(\textbf{70-181}\)} \makecell{$56$\\\(\textbf{38-84}\)} \makecell{$28$\\\(\textbf{20-39}\)} \makecell{$19$\\\(\textbf{14-25}\)} \makecell{$11.3$\\\(\textbf{8.8-14.4}\)} \makecell{$3.76$\\\(\textbf{3.29-4.29}\)}
    \makecell{type of\\processing} fit                
    \makecell{control:\\no correction} \makecell{1 species} \colorbox[HTML]{d55e00}{\makecell{$58.92$}} \colorbox[HTML]{009e73}{\makecell{$142.83$}} \colorbox[HTML]{009e73}{\makecell{$80.93$}} \colorbox[HTML]{d55e00}{\makecell{$145.00$}} \colorbox[HTML]{009e73}{\makecell{$27.55$}} \colorbox[HTML]{d55e00}{\makecell{$54.11$}} \colorbox[HTML]{d55e00}{\makecell{$110.39$}} \colorbox[HTML]{d55e00}{\makecell{$287.87$}}
      \makecell{2 - fast sp.\\2 - slow sp.} \colorbox[HTML]{009e73}{\makecell{$15.85$\\$22… \colorbox[HTML]{d55e00}{\makecell{$70.02$\\$44… \colorbox[HTML]{009e73}{\makecell{$12.82$\\$13… \colorbox[HTML]{009e73}{\makecell{$53.66$\\$44… \colorbox[HTML]{d55e00}{\makecell{$13.17$\\$38… \colorbox[HTML]{009e73}{\makecell{$16.68$\\$42… \colorbox[HTML]{d55e00}{\makecell{$47.23$\\$28… \colorbox[HTML]{d55e00}{\makecell{$16.59$\\$64…
    \makecell{new method:\\cut and shift} \makecell{1 species} \colorbox[HTML]{009e73}{\makecell{$161.43$}} \colorbox[HTML]{009e73}{\makecell{$130.03$}} \colorbox[HTML]{009e73}{\makecell{$100.67$}} \colorbox[HTML]{009e73}{\makecell{$53.11$}} \colorbox[HTML]{009e73}{\makecell{$26.25$}} \colorbox[HTML]{009e73}{\makecell{$17.43$}} \colorbox[HTML]{009e73}{\makecell{$11.52$}} \colorbox[HTML]{009e73}{\makecell{$3.62$}}
      \makecell{2 - fast sp.\\2 - slow sp.} \colorbox[HTML]{009e73}{\makecell{$120.86$\\$4… \colorbox[HTML]{009e73}{\makecell{$4.33$\\$131… \colorbox[HTML]{009e73}{\makecell{$100.67$\\$1… \colorbox[HTML]{009e73}{\makecell{$0.00$\\$53…. \colorbox[HTML]{009e73}{\makecell{$0.01$\\$26…. \colorbox[HTML]{009e73}{\makecell{$17.43$}} \colorbox[HTML]{009e73}{\makecell{$11.52$}} \colorbox[HTML]{009e73}{\makecell{$3.62$\\$3.6…
    \makecell{old method:\\weight=0} \makecell{1 species} \colorbox[HTML]{d55e00}{\makecell{$383.41$}} \colorbox[HTML]{d55e00}{\makecell{$253.55$}} \colorbox[HTML]{d55e00}{\makecell{$293.12$}} \colorbox[HTML]{d55e00}{\makecell{$301.92$}} \colorbox[HTML]{d55e00}{\makecell{$287.00$}} \colorbox[HTML]{d55e00}{\makecell{$143.32$}} \colorbox[HTML]{d55e00}{\makecell{$142.88$}} \colorbox[HTML]{d55e00}{\makecell{$362.80$}}
      \makecell{2 - fast sp.\\2 - slow sp.} \colorbox[HTML]{d55e00}{\makecell{$28.66$\\$81… \colorbox[HTML]{d55e00}{\makecell{$45.97$\\$11… \colorbox[HTML]{d55e00}{\makecell{$36.14$\\$72… \colorbox[HTML]{009e73}{\makecell{$44.09$\\$17… \colorbox[HTML]{d55e00}{\makecell{$19.30$\\$74… \colorbox[HTML]{d55e00}{\makecell{$10.94$\\$21… \colorbox[HTML]{d55e00}{\makecell{$34.31$\\$10… \colorbox[HTML]{d55e00}{\makecell{$20.37$\\$77…
  • I copied the latex code in the latex online editor Overleaf and compiled it there. The rendered PDF looks like this for the beamer slide:
  • and here the rendered PDF of a long table

2.6.12 Experiment 3: Run models on experimental data

2.6.12.1 node 1 (overview)
  • this is the prepare-jupyter function
    %cd /beegfs/ye53nis/drmed-git
    import logging
    import os
    import sys
    
    import matplotlib.pyplot as plt
    import numpy as np
    import pandas as pd
    import seaborn as sns
    
    from pathlib import Path
    from pprint import pprint
    from tensorflow.keras.optimizers import Adam
    from mlflow.keras import load_model
    
    FLUOTRACIFY_PATH = '/beegfs/ye53nis/drmed-git/src/'
    sys.path.append(FLUOTRACIFY_PATH)
    from fluotracify.applications import corr_fit_object as cfo
    from fluotracify.training import build_model as bm
    
    data_path = Path(data_path)
    output_path = Path(output_path)
    log_path = output_path.parent / f'{output_path.name}.log'
    
    logging.basicConfig(filename=log_path,
                        filemode='w', format='%(asctime)s - %(message)s',
                        force=True)
    
    log = logging.getLogger(__name__)
    log.setLevel(logging.DEBUG)
    
    sns.set_theme(style="whitegrid", font_scale=2, palette='colorblind',
                  context='paper')
    class ParameterClass():
        """Stores parameters for correlation """
        def __init__(self):
            # Where the data is stored.
            self.data = []
            self.objectRef = []
            self.subObjectRef = []
            self.colors = ['blue', 'green', 'red', 'cyan', 'magenta',
                           'yellow', 'black']
            self.numOfLoaded = 0
            # very fast from Ncasc ~ 14 onwards
            self.NcascStart = 0
            self.NcascEnd = 30  # 25
            self.Nsub = 6  # 6
            self.photonLifetimeBin = 10  # used for photon decay
            self.photonCountBin = 1  # used for time series
    
    par_obj = ParameterClass()
    
    model_ls = ['ff67be0b68e540a9a29a36a2d0c7a5be', '347669d050f344ad9fb9e480c814f727',
                '714af8cd12c1441eac4ca980e8c20070', '34a6d207ac594035b1009c330fb67a65',
                '484af471c61943fa90e5f78e78a229f0', '0cd2023eeaf745aca0d3e8ad5e1fc653',
                'fe81d71c52404ed790b3a32051258da9', '19e3e786e1bc4e2b93856f5dc9de8216',
                'c1204e3a8a1e4c40a35b5b7b1922d1ce']
    
    model_name_ls = [f'{s:.5}' for s in model_ls]
    
    # scaler_ls = ['minmax', 'robust', 'maxabs', 'l2', 'standard', 'quant_g', 'standard',
    #              'standard', 'robust']
    
    pred_thresh = 0.5
    
    if data_path.name == "1911DD_atto+LUVs":
        path_clean1 = data_path / 'clean_ptu_part1/'
        path_clean2 = data_path / 'clean_ptu_part2/'
        path_dirty1 = data_path / 'dirty_ptu_part1/'
        path_dirty2 = data_path / 'dirty_ptu_part2/'
        files_clean1 = [path_clean1 / f for f in os.listdir(path_clean1) if f.endswith('.ptu')]
        files_clean2 = [path_clean2 / f for f in os.listdir(path_clean2) if f.endswith('.ptu')]
        files_dirty1 = [path_dirty1 / f for f in os.listdir(path_dirty1) if f.endswith('.ptu')]
        files_dirty2 = [path_dirty2 / f for f in os.listdir(path_dirty2) if f.endswith('.ptu')]
    
    if data_path.name == "191113_Pex5_2_structured":
        path_clean = data_path / 'HsPEX5EGFP 1-100001'
        path_dirty = data_path / 'TbPEX5EGFP 1-10002'
        files_clean = [path_clean / f for f in os.listdir(path_clean) if f.endswith('.ptu')]
        files_dirty = [path_dirty / f for f in os.listdir(path_dirty) if f.endswith('.ptu')]
    
    def predict_correct_correlate_ptu(files, model_id, method, out_path):
    
        logged_scaler = Path(f'/beegfs/ye53nis/drmed-git/data/mlruns/10/{model_ls[model_id]}/params/scaler')
        logged_scaler = !cat $logged_scaler
        logged_scaler = logged_scaler[0]
    
        logged_model = Path(f'/beegfs/ye53nis/drmed-git/data/mlruns/10/{model_ls[model_id]}/artifacts/model')
        logged_model = load_model(logged_model, compile=False)
        logged_model.compile(loss=bm.binary_ce_dice_loss(),
                             optimizer=Adam(),
                             metrics = bm.unet_metrics([0.1, 0.3, 0.5, 0.7, 0.9]))
        if method == 'delete_and_shift':
            method_str = 'DELSHIFT'
        elif method == 'delete':
            method_str = 'DEL'
        for idx, myfile in enumerate(files):
            ptufile = cfo.PicoObject(myfile, par_obj)
            ptufile.predictTimeSeries(model=logged_model,
                                      scaler=logged_scaler)
            ptufile.correctTCSPC(method=method)
            for key in list(ptufile.trueTimeArr.keys()):
                if method_str in key:
                    ptufile.get_autocorrelation(method='tttr2xfcs', name=key)
    
    
            for m in ['multipletau', 'tttr2xfcs', 'tttr2xfcs_with_weights']:
                if m in list(ptufile.autoNorm.keys()):
                    for key, item in list(ptufile.autoNorm[m].items()):
                        if method_str in key:
                            ptufile.save_autocorrelation(name=key, method=m,
                                                         output_path=out_path)
    
    
    def correlate_ptu(files, out_path):
        for idx, myfile in enumerate(files):
            ptufile = cfo.PicoObject(myfile, par_obj)
            for key in list(ptufile.trueTimeArr.keys()):
                ptufile.get_autocorrelation(method='tttr2xfcs', name=key)
    
    
            for m in ['multipletau', 'tttr2xfcs', 'tttr2xfcs_with_weights']:
                if m in list(ptufile.autoNorm.keys()):
                    for key, item in list(ptufile.autoNorm[m].items()):
                        ptufile.save_autocorrelation(name=key, method=m,
                                                     output_path=out_path)
    
    
    
    /beegfs/ye53nis/drmed-git
    2023-01-10 17:38:18.004165: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2023-01-10 17:38:18.004236: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    
  • this is the kill-jupyter function
    os._exit(00)
    
    7a752b61-60c1-4322-8105-e8dbd705caa0
    
  • now call the model and apply it to af488+luv data (with peak artifacts), first with delete_and_shift, which we later called cut and stitch
    model_id = 0
    
    out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
    
    os.makedirs(out_dir, exist_ok=True)
    
    predict_correct_correlate_ptu(
        files=files_dirty1,
        model_id=model_id, method='delete_and_shift',
        out_path=out_dir)
    
    2022-05-23 14:47:32.428742: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2022-05-23 14:47:32.428785: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
    2022-05-23 14:47:32.428823: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node105): /proc/driver/nvidia/version does not exist
    2022-05-23 14:47:32.429138: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    
    2df4455b-0130-4b59-af50-3a85f1419fd2
    
    /beegfs/ye53nis/drmed-git
    2022-05-23 19:52:48.412017: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2022-05-23 19:52:48.412054: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    
    model_id = 0
    
    out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
    
    os.makedirs(out_dir, exist_ok=True)
    
    predict_correct_correlate_ptu(
        files=files_dirty2,
        model_id=model_id, method='delete_and_shift',
        out_path=out_dir)
    
    2022-05-23 19:53:11.911961: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2022-05-23 19:53:11.912000: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
    2022-05-23 19:53:11.912024: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node105): /proc/driver/nvidia/version does not exist
    2022-05-23 19:53:11.912392: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    
    • on node 2:
      model_id = 1
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty1 and files_dirty2,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
    • on node 3:
      model_id = 2
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty1 and files_dirty2,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
    • on node 2:
      model_id = 3  # 34a6d
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty,
          model_id=0,
          out_path=out_dir)
      
    • on node 3:
      model_id = 4  # 484af
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty,
          model_id=0,
          out_path=out_dir)
      
      ea569473-6848-41cb-bae6-4f5ac1590e71
      
      /beegfs/ye53nis/drmed-git
      2022-05-23 21:51:34.834624: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-05-23 21:51:34.834663: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
    • on node 1:
      model_id = 5 # 0cd20
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
      files=files_dirty1,
      model_id=model_id, method='delete_and_shift',
      out_path=out_dir)
      
      2022-05-23 21:52:02.797179: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-05-23 21:52:02.797239: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-05-23 21:52:02.797260: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node105): /proc/driver/nvidia/version does not exist
      2022-05-23 21:52:02.797625: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      /beegfs/ye53nis/drmed-git
      2022-05-24 10:12:54.289832: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-05-24 10:12:54.289868: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 5 # 0cd20
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
      files=files_dirty2,
      model_id=model_id, method='delete_and_shift',
      out_path=out_dir)
      
      2022-05-24 10:13:01.303436: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-05-24 10:13:01.303503: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-05-24 10:13:01.303541: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node105): /proc/driver/nvidia/version does not exist
      2022-05-24 10:13:01.303975: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
    • on node 2:
      model_id = 6 # fe81d
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
      files=files_dirty1,
      model_id=model_id, method='delete_and_shift',
      out_path=out_dir)
      
      bf3b2ade-cc27-404c-b7bc-9db7a0583d4e
      
      /beegfs/ye53nis/drmed-git
      2022-05-24 13:29:48.438857: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-05-24 13:29:48.438908: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
    • on node 1:
      model_id = 7 # 19e3e
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
      files=files_dirty1,
      model_id=model_id, method='delete_and_shift',
      out_path=out_dir)
      
      2022-05-24 13:30:26.419948: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-05-24 13:30:26.420019: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-05-24 13:30:26.420055: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node105): /proc/driver/nvidia/version does not exist
      2022-05-24 13:30:26.420612: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      c55ec613-2fe9-4dd8-bbb0-03251a8173c2
      
      /beegfs/ye53nis/drmed-git
      2022-05-24 17:07:04.399895: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-05-24 17:07:04.399949: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
    • on node 1:
      model_id = 7 # 19e3e
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
      files=files_dirty2,
      model_id=model_id, method='delete_and_shift',
      out_path=out_dir)
      
      2022-05-24 17:07:10.270368: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-05-24 17:07:10.270428: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-05-24 17:07:10.270454: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node105): /proc/driver/nvidia/version does not exist
      2022-05-24 17:07:10.270792: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
    • on node 3:
      model_id = 8  # c1204
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
  • now do af488 and af488+luvs correlations without any correction
    • on node 3:
      out_dir = output_path / f'clean/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      correlate_ptu(
          files=files_clean_1 and files_clean2,
          out_path=out_dir)
      
    • on node 1:
      a26d0ea2-4340-4437-afb6-5e931c8efe68
      
      /beegfs/ye53nis/drmed-git
      2022-05-27 13:21:04.304353: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-05-27 13:21:04.304423: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      out_dir = output_path / f'dirty/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      correlate_ptu(
          files=files_dirty2,
          out_path=out_dir)
      
    • on node 3:
      out_dir = output_path / f'clean/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      correlate_ptu(
          files=files_clean_1 and files_clean2,
          out_path=out_dir)
      
      out_dir = output_path / f'dirty/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      correlate_ptu(
          files=files_dirty2,
          out_path=out_dir)
      
  • now call the model and apply it to af488 data (without peak artifacts - this is a control, the models should not detect peak artifacts here), first with delete_and_shift, which we later called cut and stitch
    • on node 1:
      model_id = 5 # 0cd20
      
      out_dir = output_path / f'clean_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
      files=files_clean1,
      model_id=model_id, method='delete_and_shift',
      out_path=out_dir)
      
      2022-05-25 14:48:29.187895: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-05-25 14:48:29.187952: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-05-25 14:48:29.187985: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node105): /proc/driver/nvidia/version does not exist
      2022-05-25 14:48:29.188422: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      7e9aa800-f60f-4d9a-b565-722c72070338
      
      /beegfs/ye53nis/drmed-git
      2022-05-25 16:48:58.124680: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-05-25 16:48:58.124720: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 5 # 0cd20
      
      out_dir = output_path / f'clean_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
      files=files_clean2,
      model_id=model_id, method='delete_and_shift',
      out_path=out_dir)
      
      2022-05-25 16:49:18.365951: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-05-25 16:49:18.366011: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-05-25 16:49:18.366045: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node105): /proc/driver/nvidia/version does not exist
      2022-05-25 16:49:18.366534: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
    • on node 3:
      model_id = 2 # 714af
      
      out_dir = output_path / f'clean_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_clean_1 and files_clean2,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
    • on node 2:
      model_id = 0 # ff67b
      
      out_dir = output_path / f'clean_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_clean1,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
    • on node 1:
      model_id = 1 # 34766
      
      out_dir = output_path / f'clean_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
      files=files_clean1,
      model_id=model_id, method='delete_and_shift',
      out_path=out_dir)
      
      2022-05-26 00:11:02.988574: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-05-26 00:11:02.988629: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-05-26 00:11:02.988653: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node105): /proc/driver/nvidia/version does not exist
      2022-05-26 00:11:02.988969: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      /beegfs/ye53nis/drmed-git
      2022-05-26 11:49:10.130005: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-05-26 11:49:10.130041: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 1
      
      out_dir = output_path / f'clean_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
      files=files_clean2,
      model_id=model_id, method='delete_and_shift',
      out_path=out_dir)
      
      2022-05-26 11:49:16.690889: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-05-26 11:49:16.690951: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-05-26 11:49:16.690989: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node105): /proc/driver/nvidia/version does not exist
      2022-05-26 11:49:16.691462: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
    • on node 2:
      model_id = 3 # 34a6d
      
      out_dir = output_path / f'clean_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_clean_1 and files_clean2,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
    • on node 1:
      a26d0ea2-4340-4437-afb6-5e931c8efe68
      
      /beegfs/ye53nis/drmed-git
      2022-05-27 13:21:04.304353: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-05-27 13:21:04.304423: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
    • on node 2:
      model_id = 7 # 19e3e
      
      out_dir = output_path / f'clean_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_clean_1 and files_clean2,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
    • on node 3:
      model_id = 4 # 484af
      
      out_dir = output_path / f'clean_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_clean_1 and files_clean2,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
    • on node 2:
      model_id = 8 # c1204
      
      out_dir = output_path / f'clean_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_clean_1 and files_clean2,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
    • on node 3:
      model_id = 6 # fe81d
      
      out_dir = output_path / f'clean_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_clean_1 and files_clean2,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
  • now call the model and apply it to pex5 data. First the correlations of Trypanosoma brucei-PEX5-eGFP (peak artifacts) and Homo sapiens-PEX5-eGFP (no artifacts) without corrections. Not that prepare-jupyter now looks like this: #+CALL: prepare-jupyter("/beegfs/ye53nis/data/191113_Pex5_2_structured", output_path="/beegfs/ye53nis/drmed-git/data/exp-220227-unet/2022-06-02_experimental-pex5/")
    • on node 1:
      /beegfs/ye53nis/drmed-git
      
      out_dir = output_path / f'clean/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      correlate_ptu(
          files=files_clean,
          out_path=out_dir)
      
      274c16ef-1bb8-47fd-9ac9-d83e193d2c7c
      
      /beegfs/ye53nis/drmed-git
      2022-06-03 00:20:04.405561: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-06-03 00:20:04.405614: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      out_dir = output_path / f'dirty/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      correlate_ptu(
          files=files_dirty,
          out_path=out_dir)
      
      2022-06-02 16:25:02.408577: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-06-02 16:25:02.408633: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-06-02 16:25:02.408668: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node034): /proc/driver/nvidia/version does not exist
      2022-06-02 16:25:02.409190: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
  • now call the model and apply it to all pex5 data (with and without peak artifacts), first with delete_and_shift correction, which we later called cut and stitch
    • on node 2:
      model_id = 0 # ff67b
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty and files_clean,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
    • on node 3:
      model_id = 1  # 34766
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty and files_clean,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      e17bafa1-336e-4ae4-9eac-2e4e754b163b
      
      /beegfs/ye53nis/drmed-git
      2022-06-03 00:35:48.794905: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-06-03 00:35:48.794949: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
    • on node 1:
      model_id = 2  # 714af
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
      2022-06-03 00:36:59.208902: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-06-03 00:36:59.208990: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-06-03 00:36:59.209034: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node034): /proc/driver/nvidia/version does not exist
      2022-06-03 00:36:59.209615: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      cb8ca215-2cec-4c8a-b57f-9e7a3fa0850a
      
      /beegfs/ye53nis/drmed-git
      2022-06-03 01:03:42.934510: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-06-03 01:03:42.934560: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 2  # 714af
      
      out_dir = output_path / f'clean_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_clean,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
      2022-06-03 01:03:53.314614: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-06-03 01:03:53.314706: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-06-03 01:03:53.314759: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node034): /proc/driver/nvidia/version does not exist
      2022-06-03 01:03:53.315343: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
    • on node 2:
      model_id = 3 # 34a6d
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty and files_clean,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
    • on node 1:
      model_id = 4  # 484af
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
      2022-06-03 16:05:25.702609: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-06-03 16:05:25.702662: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-06-03 16:05:25.702697: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node034): /proc/driver/nvidia/version does not exist
      2022-06-03 16:05:25.703121: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      /beegfs/ye53nis/drmed-git
      2022-06-03 17:13:28.754363: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-06-03 17:13:28.754407: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 4  # 484af
      
      out_dir = output_path / f'clean_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_clean,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
      2022-06-03 17:13:35.380945: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-06-03 17:13:35.380992: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-06-03 17:13:35.381021: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node034): /proc/driver/nvidia/version does not exist
      2022-06-03 17:13:35.381294: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
    • on node 3:
      model_id = 5  # 0cd20
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty and files_clean,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
    • on node 2:
      model_id = 6 # fe81d
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty and files_clean,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
    • on node 1:
      model_id = 7  # 19e3e
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
      2022-06-03 17:59:22.958003: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-06-03 17:59:22.958066: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-06-03 17:59:22.958106: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node034): /proc/driver/nvidia/version does not exist
      2022-06-03 17:59:22.958596: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      /beegfs/ye53nis/drmed-git
      2022-06-03 18:12:09.643502: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-06-03 18:12:09.643564: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 7  # 19e3e
      
      out_dir = output_path / f'clean_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_clean,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
      2022-06-03 18:12:16.305111: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-06-03 18:12:16.305153: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-06-03 18:12:16.305181: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node034): /proc/driver/nvidia/version does not exist
      2022-06-03 18:12:16.305464: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      

      on node 3:

      model_id = 8  # c1204
      
      out_dir = output_path / f'dirty_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty and files_clean,
          model_id=model_id, method='delete_and_shift',
          out_path=out_dir)
      
  • now call the model and apply it to all pex5 data (with and without peak artifacts), now with delete correction, which we later called set to zero
    • on node 2
      model_id = 0 # ff67b
      
      out_dir = output_path / f'dirty_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty and files_clean,
          model_id=model_id, method='delete',
          out_path=out_dir)
      

      on node 1:

      model_id = 1  # 34766
      
      out_dir = output_path / f'dirty_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty and files_clean,
          model_id=model_id,
          method='delete',
          out_path=out_dir)
      
      2022-06-03 18:34:51.572833: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-06-03 18:34:51.572883: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-06-03 18:34:51.572912: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node034): /proc/driver/nvidia/version does not exist
      2022-06-03 18:34:51.573277: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      947e95f9-7f45-4e35-a5f6-857c1a00df32
      
      /beegfs/ye53nis/drmed-git
      2022-06-05 19:58:02.542852: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-06-05 19:58:02.542898: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 1  # 34766
      
      out_dir = output_path / f'clean_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_clean,
          model_id=model_id,
          method='delete',
          out_path=out_dir)
      
      2022-06-05 19:58:25.596269: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-06-05 19:58:25.596330: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-06-05 19:58:25.596366: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node034): /proc/driver/nvidia/version does not exist
      2022-06-05 19:58:25.596907: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
    • on node 3
      model_id = 2  # 714af
      
      out_dir = output_path / f'dirty_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty and files_clean,
          model_id=model_id, method='delete',
          out_path=out_dir)
      
    • on node 1:
      model_id = 3  # 34a6d
      
      out_dir = output_path / f'dirty_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty and files_clean,
          model_id=model_id,
          method='delete',
          out_path=out_dir)
      
      2022-06-05 20:29:47.728848: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-06-05 20:29:47.728912: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-06-05 20:29:47.728949: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node034): /proc/driver/nvidia/version does not exist
      2022-06-05 20:29:47.729437: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      2623f23f-2abb-44a9-8a8a-37fe279cda97
      
      /beegfs/ye53nis/drmed-git
      2022-06-06 11:30:37.942306: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-06-06 11:30:37.942353: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 3  # 34a6d
      
      out_dir = output_path / f'clean_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_clean,
          model_id=model_id,
          method='delete',
          out_path=out_dir)
      
      2022-06-06 11:30:52.913876: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-06-06 11:30:52.913938: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-06-06 11:30:52.913990: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node034): /proc/driver/nvidia/version does not exist
      2022-06-06 11:30:52.914431: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
    • on node 2
      model_id = 4  # 484af
      
      out_dir = output_path / f'dirty_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty and files_clean,
          model_id=model_id, method='delete',
          out_path=out_dir)
      
    • on node 3
      model_id = 5  # 0cd20
      
      out_dir = output_path / f'dirty_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty and files_clean,
          model_id=model_id, method='delete',
          out_path=out_dir)
      
    • on node 1:
      model_id = 6   # fe81d
      
      out_dir = output_path / f'dirty_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty and files_clean,
          model_id=model_id,
          method='delete',
          out_path=out_dir)
      
      2022-06-06 16:01:59.893483: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2022-06-06 16:01:59.893526: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2022-06-06 16:01:59.893551: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node034): /proc/driver/nvidia/version does not exist
      2022-06-06 16:01:59.893887: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      ---------------------------------------------------------------------------
      NameError                                 Traceback (most recent call last)
      Input In [1], in <cell line: 1>()
      ----> 1 os._exit(00)
      
      NameError: name 'os' is not defined
      
      /beegfs/ye53nis/drmed-git
      2022-06-06 16:01:50.356512: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2022-06-06 16:01:50.356556: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 6  # fe81d
      
      out_dir = output_path / f'clean_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_clean,
          model_id=model_id,
          method='delete',
          out_path=out_dir)
      
    • on node 2
      model_id = 7  # 19e3e
      
      out_dir = output_path / f'dirty_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      predict_correct_correlate_ptu(
          files=files_dirty and files_clean,
          model_id=model_id, method='delete',
          out_path=out_dir)
      
2.6.12.2 node 2
  1. Set up tmux (if we haven't done that before) (#+CALL: setup-tmux[:session local])
         
    sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:
    > ye53nis@ara-login01.rz.uni-jena.de's password:
  2. Request compute node
    cd /
    srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
    
  3. Start Jupyter Lab (#+CALL: jpt-tmux[:session jpmux])
    conda activate tf
    export PORT=8890
    export XDG_RUNTIME_DIR=''
    export XDG_RUNTIME_DIR=""
    jupyter lab --no-browser --port=$PORT
    
    (tf) [ye53nis@node152 /]$ jupyter lab --no-browser --port=$PORT
    [I 2022-06-03 00:24:43.720 ServerApp] jupyterlab | extension was successfully linked.
    [I 2022-06-03 00:24:44.304 ServerApp] nbclassic | extension was successfully linked.
    [I 2022-06-03 00:24:44.363 ServerApp] nbclassic | extension was successfully loaded.
    [I 2022-06-03 00:24:44.364 LabApp] JupyterLab extension loaded from /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/jupyterlab
    [I 2022-06-03 00:24:44.364 LabApp] JupyterLab application directory is /home/ye53nis/.conda/envs/tf/share/jupyter/lab
    [I 2022-06-03 00:24:44.368 ServerApp] jupyterlab | extension was successfully loaded.
    [I 2022-06-03 00:24:44.369 ServerApp] Serving notebooks from local directory: /
    [I 2022-06-03 00:24:44.369 ServerApp] Jupyter Server 1.13.5 is running at:
    [I 2022-06-03 00:24:44.369 ServerApp] http://localhost:8890/lab?token=b2f94d18a55263d5a94b5c7e52c96c60bfdfc076bfb6f00a
    [I 2022-06-03 00:24:44.369 ServerApp]  or http://127.0.0.1:8890/lab?token=b2f94d18a55263d5a94b5c7e52c96c60bfdfc076bfb6f00a
    [I 2022-06-03 00:24:44.370 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
    [C 2022-06-03 00:24:44.376 ServerApp]
    
        To access the server, open this file in a browser:
            file:///home/ye53nis/.local/share/jupyter/runtime/jpserver-207633-open.html
        Or copy and paste one of these URLs:
            http://localhost:8890/lab?token=b2f94d18a55263d5a94b5c7e52c96c60bfdfc076bfb6f00a
         or http://127.0.0.1:8890/lab?token=b2f94d18a55263d5a94b5c7e52c96c60bfdfc076bfb6f00a
    
  4. Create SSH Tunnel for jupyter lab to the local computer (e.g. #+CALL: ssh-tunnel(port="8889", node="node160"))
                     
    sh-5.1$ sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:          
    ye53nis@node152's password:              
    Last login: Mon Jun 6 11:31:14 2022 from login01.ara
  5. this subtree was connected to another compute node. I kept the going track record of all executed commands in the mother node. These following two code blocks are just examples. The process was the same as in the mother node.
    /beegfs/ye53nis/drmed-git
    2022-06-06 16:03:48.742170: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2022-06-06 16:03:48.742202: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    
    model_id = 7  # 19e3e
    
    out_dir = output_path / f'dirty_delete_{model_name_ls[model_id]}/'
    
    os.makedirs(out_dir, exist_ok=True)
    
    predict_correct_correlate_ptu(
        files=files_dirty,
        model_id=model_id,
        method='delete',
        out_path=out_dir)
    
    2022-06-06 16:03:54.537136: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2022-06-06 16:03:54.537186: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
    2022-06-06 16:03:54.537212: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node152): /proc/driver/nvidia/version does not exist
    2022-06-06 16:03:54.537595: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    
    744ddb63-44d8-4041-b531-2ce90bcb0e5c
    
    /beegfs/ye53nis/drmed-git
    2022-06-06 11:31:31.271711: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2022-06-06 11:31:31.271745: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    
    model_id = 4  # 484af
    
    out_dir = output_path / f'clean_delete_{model_name_ls[model_id]}/'
    
    os.makedirs(out_dir, exist_ok=True)
    
    predict_correct_correlate_ptu(
        files=files_clean,
        model_id=model_id,
        method='delete',
        out_path=out_dir)
    
    2022-06-06 11:31:39.219733: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2022-06-06 11:31:39.219776: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
    2022-06-06 11:31:39.219797: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node152): /proc/driver/nvidia/version does not exist
    2022-06-06 11:31:39.220117: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `
    
2.6.12.3 node 3
  1. Set up tmux (if we haven't done that before) (#+CALL: setup-tmux[:session local])
         
    sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:
    > ye53nis@ara-login01.rz.uni-jena.de's password:
  2. Request compute node
    cd /
    srun -p s_standard --time=7-10:00:00 --ntasks-per-node=24 --mem-per-cpu=2000 --pty bash
    
  3. Start Jupyter Lab (#+CALL: jpt-tmux[:session jpmux])
    conda activate tf
    export PORT=8891
    export XDG_RUNTIME_DIR=''
    export XDG_RUNTIME_DIR=""
    jupyter lab --no-browser --port=$PORT
    
    (tf) [ye53nis@node155 /]$ jupyter lab --no-browser --port=$PORT
    [I 2022-06-03 00:28:12.112 ServerApp] jupyterlab | extension was successfully linked.
    [I 2022-06-03 00:28:12.735 ServerApp] nbclassic | extension was successfully linked.
    [I 2022-06-03 00:28:12.778 ServerApp] nbclassic | extension was successfully loaded.
    [I 2022-06-03 00:28:12.780 LabApp] JupyterLab extension loaded from /home/ye53nis/.conda/envs/tf/lib/python3.9/site-packages/jupyterlab
    [I 2022-06-03 00:28:12.780 LabApp] JupyterLab application directory is /home/ye53nis/.conda/envs/tf/share/jupyter/lab
    [I 2022-06-03 00:28:12.783 ServerApp] jupyterlab | extension was successfully loaded.
    [I 2022-06-03 00:28:12.784 ServerApp] Serving notebooks from local directory: /
    [I 2022-06-03 00:28:12.784 ServerApp] Jupyter Server 1.13.5 is running at:
    [I 2022-06-03 00:28:12.784 ServerApp] http://localhost:8891/lab?token=fe7d23f6a124fa20ea2b5d7f0dfadc3b20f0db64f8feadfd
    [I 2022-06-03 00:28:12.784 ServerApp]  or http://127.0.0.1:8891/lab?token=fe7d23f6a124fa20ea2b5d7f0dfadc3b20f0db64f8feadfd
    [I 2022-06-03 00:28:12.784 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
    [C 2022-06-03 00:28:12.790 ServerApp]
    
        To access the server, open this file in a browser:
            file:///home/ye53nis/.local/share/jupyter/runtime/jpserver-12523-open.html
        Or copy and paste one of these URLs:
            http://localhost:8891/lab?token=fe7d23f6a124fa20ea2b5d7f0dfadc3b20f0db64f8feadfd
         or http://127.0.0.1:8891/lab?token=fe7d23f6a124fa20ea2b5d7f0dfadc3b20f0db64f8feadfd
    
  4. Create SSH Tunnel for jupyter lab to the local computer (e.g. #+CALL: ssh-tunnel(port="8889", node="node160"))
                     
    sh-5.1$ sh-5.1$ ye53nis@ara-login01.rz.uni-jena.de's password:          
    ye53nis@node155's password:              
    Last login: Mon Jun 6 11:32:59 2022 from login01.ara
  5. this subtree was connected to another compute node. I kept the going track record of all executed commands in the mother node. These following two code blocks are just examples. The process was the same as in the mother node.
    /beegfs/ye53nis/drmed-git
    2022-06-06 16:04:50.862041: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
    2022-06-06 16:04:50.862080: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    
    model_id = 5  # 0cd20
    
    out_dir = output_path / f'dirty_delete_{model_name_ls[model_id]}/'
    
    os.makedirs(out_dir, exist_ok=True)
    
    predict_correct_correlate_ptu(
        files=files_dirty,
        model_id=model_id,
        method='delete',
        out_path=out_dir)
    
    2022-06-06 11:33:30.480931: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2022-06-06 11:33:30.480975: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
    2022-06-06 11:33:30.481005: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node155): /proc/driver/nvidia/version does not exist
    2022-06-06 11:33:30.481315: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    
    /beegfs/ye53nis/drmed-git
    
    model_id = 5  # 0cd20
    
    out_dir = output_path / f'clean_delete_{model_name_ls[model_id]}/'
    
    os.makedirs(out_dir, exist_ok=True)
    
    predict_correct_correlate_ptu(
        files=files_clean,
        model_id=model_id,
        method='delete',
        out_path=out_dir)
    
    2022-06-06 16:04:56.835044: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
    2022-06-06 16:04:56.835085: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
    2022-06-06 16:04:56.835105: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node155): /proc/driver/nvidia/version does not exist
    2022-06-06 16:04:56.835386: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
    To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
    
2.6.12.4 <2023-01-10 Di> Apply averaging and settozero correction with 0cd20
  • let's first prepare the function which loads modules, the files list, and provides a wrapper for the fluotracify correlation and correction functions:
  • prepare-jupyter-averaging(data_path="/beegfs/ye53nis/data/1911DD_atto+LUVs", output_path="/beegfs/ye53nis/drmed-git/data/exp-220227-unet/2023-01-10_experimental-averaging-delete/") prepare-jupyter-averaging(data_path="/beegfs/ye53nis/data/1911DD_atto+LUVs", output_path="/beegfs/ye53nis/drmed-git/data/exp-220227-unet/2023-01-10_experimental-averaging-delete/")
    %cd /beegfs/ye53nis/drmed-git
    import logging
    import os
    import sys
    
    import matplotlib.pyplot as plt
    import numpy as np
    import pandas as pd
    import seaborn as sns
    
    from pathlib import Path
    from pprint import pprint
    from tensorflow.keras.optimizers import Adam
    from mlflow.keras import load_model
    
    FLUOTRACIFY_PATH = '/beegfs/ye53nis/drmed-git/src/'
    sys.path.append(FLUOTRACIFY_PATH)
    from fluotracify.applications import corr_fit_object as cfo
    from fluotracify.training import build_model as bm
    
    data_path = Path(data_path)
    output_path = Path(output_path)
    log_path = output_path.parent / f'{output_path.name}.log'
    
    logging.basicConfig(filename=log_path,
                        filemode='w', format='%(asctime)s - %(message)s',
                        force=True)
    
    log = logging.getLogger(__name__)
    log.setLevel(logging.DEBUG)
    
    sns.set_theme(style="whitegrid", font_scale=2, palette='colorblind',
                  context='paper')
    
    class ParameterClass():
        """Stores parameters for correlation """
        def __init__(self):
            # Where the data is stored.
            self.data = []
            self.objectRef = []
            self.subObjectRef = []
            self.colors = ['blue', 'green', 'red', 'cyan', 'magenta',
                           'yellow', 'black']
            self.numOfLoaded = 0
            # very fast from Ncasc ~ 14 onwards
            self.NcascStart = 0
            self.NcascEnd = 30  # 25
            self.Nsub = 6  # 6
            self.photonLifetimeBin = 10  # used for photon decay
            self.photonCountBin = 1  # used for time series
    
    par_obj = ParameterClass()
    
    model_ls = ['ff67be0b68e540a9a29a36a2d0c7a5be', '347669d050f344ad9fb9e480c814f727',
                '714af8cd12c1441eac4ca980e8c20070', '34a6d207ac594035b1009c330fb67a65',
                '484af471c61943fa90e5f78e78a229f0', '0cd2023eeaf745aca0d3e8ad5e1fc653',
                'fe81d71c52404ed790b3a32051258da9', '19e3e786e1bc4e2b93856f5dc9de8216',
                'c1204e3a8a1e4c40a35b5b7b1922d1ce']
    
    model_name_ls = [f'{s:.5}' for s in model_ls]
    
    # scaler_ls = ['minmax', 'robust', 'maxabs', 'l2', 'standard', 'quant_g', 'standard',
    #              'standard', 'robust']
    
    pred_thresh = 0.5
    
    if data_path.name == "1911DD_atto+LUVs":
        path_clean1 = data_path / 'clean_ptu_part1/'
        path_clean2 = data_path / 'clean_ptu_part2/'
        path_dirty1 = data_path / 'dirty_ptu_part1/'
        path_dirty2 = data_path / 'dirty_ptu_part2/'
        files_clean1 = [path_clean1 / f for f in os.listdir(path_clean1) if f.endswith('.ptu')]
        files_clean2 = [path_clean2 / f for f in os.listdir(path_clean2) if f.endswith('.ptu')]
        files_dirty1 = [path_dirty1 / f for f in os.listdir(path_dirty1) if f.endswith('.ptu')]
        files_dirty2 = [path_dirty2 / f for f in os.listdir(path_dirty2) if f.endswith('.ptu')]
    
    if data_path.name == "191113_Pex5_2_structured":
        path_clean = data_path / 'HsPEX5EGFP 1-100001'
        path_dirty = data_path / 'TbPEX5EGFP 1-10002'
        files_clean = [path_clean / f for f in os.listdir(path_clean) if f.endswith('.ptu')]
        files_dirty = [path_dirty / f for f in os.listdir(path_dirty) if f.endswith('.ptu')]
    
    def predict_correct_correlate_ptu(files, model_id, method, out_path):
    
        logged_scaler = Path(f'/beegfs/ye53nis/drmed-git/data/mlruns/10/{model_ls[model_id]}/params/scaler')
        logged_scaler = !cat $logged_scaler
        logged_scaler = logged_scaler[0]
    
        logged_model = Path(f'/beegfs/ye53nis/drmed-git/data/mlruns/10/{model_ls[model_id]}/artifacts/model')
        logged_model = load_model(logged_model, compile=False)
        logged_model.compile(loss=bm.binary_ce_dice_loss(),
                             optimizer=Adam(),
                             metrics = bm.unet_metrics([0.1, 0.3, 0.5, 0.7, 0.9]))
        if method == 'delete_and_shift':
            method_corr = 'tttr2xfcs'
            method_str = 'DELSHIFT'
        elif method == 'delete':
            method_corr = 'tttr2xfcs'
            method_str = 'DEL'
        elif method == 'weights':
            method_corr = 'tttr2xfcs_with_weights'
            method_str = 'tttr2xfcs_with_weights'
        elif method == 'averaging':
            method_corr = 'tttr2xfcs_with_averaging'
            method_str = 'tttr2xfcs_with_averaging'
        for idx, myfile in enumerate(files):
            ptufile = cfo.PicoObject(myfile, par_obj)
            ptufile.predictTimeSeries(method='unet',
                                      model=logged_model,
                                      scaler=logged_scaler)
            ptufile.correctTCSPC(method=method)
            if method in ['delete', 'delete_and_shift', 'weights']:
                for key in ptufile.trueTimeArr.keys():
                    ptufile.get_autocorrelation(method=method_corr, name=key)
            elif method == 'averaging':
                for key in ptufile.trueTimeParts.keys():
                    ptufile.get_autocorrelation(method=method_corr, name=key)
    
            if method_corr in list(ptufile.autoNorm.keys()):
                for key in list(ptufile.autoNorm[method_corr].keys()):
                    if ((method_str in key) or
                        (method_str in ['tttr2xfcs_with_weights',
                                        'tttr2xfcs_with_averaging'])):
                        ptufile.save_autocorrelation(name=key, method=method_corr,
                                                     output_path=out_path)
    
    /beegfs/ye53nis/drmed-git
    
    import importlib
    importlib.reload(cfo)
    
    <module 'fluotracify.applications.corr_fit_object' from '/beegfs/ye53nis/drmed-git/src/fluotracify/applications/corr_fit_object.py'>
    
  • let's also define a function to kill the jupyter environment. This is due to the fluotracify correlation algorithm memory allocation problem I haven't figured out yet. It will fill the memory after processing ~300 files of the af488 experiments (higher count rates). Because I haven't figured out how to solve this problem, I split the folders in parts and restart the environment.
    os._exit(00)
    
    7a752b61-60c1-4322-8105-e8dbd705caa0
    
  • first: af488luvs and af488 with averaging and delete correction (later we called it set to zero). We only use mode 0cd20 for prediction, because this has proved to be the most robust.
    • #+CALL: prepare-jupyter-averaging()
      /beegfs/ye53nis/drmed-git
      2023-01-12 13:29:49.334418: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2023-01-12 13:29:49.334472: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 5
      
      out_dir = output_path / f'dirty_averaging_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      ptufile = predict_correct_correlate_ptu(
          files=files_dirty1,
          model_id=model_id,
          method='averaging',
          out_path=out_dir)
      
    • #+CALL: kill-jupyter()
      1603e9bd-2d3a-4295-ace3-d86b1cdf4f4f
      
    • #+CALL: prepare-jupyter-averaging() etc…
      /beegfs/ye53nis/drmed-git
      2023-01-12 17:28:27.364198: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2023-01-12 17:28:27.364273: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 5
      
      out_dir = output_path / f'dirty_averaging_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      ptufile = predict_correct_correlate_ptu(
          files=files_dirty2,
          model_id=model_id,
          method='averaging',
          out_path=out_dir)
      
      2023-01-12 17:28:58.975546: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2023-01-12 17:28:58.975606: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2023-01-12 17:28:58.975643: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node117): /proc/driver/nvidia/version does not exist
      2023-01-12 17:28:58.976216: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      /beegfs/ye53nis/drmed-git
      2023-01-13 12:01:00.676480: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2023-01-13 12:01:00.676539: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 5
      
      out_dir = output_path / f'clean_averaging_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      ptufile = predict_correct_correlate_ptu(
          files=files_clean1,
          model_id=model_id,
          method='averaging',
          out_path=out_dir)
      
      2023-01-13 12:07:29.082697: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2023-01-13 12:07:29.082756: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2023-01-13 12:07:29.082791: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node117): /proc/driver/nvidia/version does not exist
      2023-01-13 12:07:29.083227: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      5b633a1b-0c54-453f-a4bf-a9d30182fdd3
      
      /beegfs/ye53nis/drmed-git
      2023-01-13 14:38:58.893878: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2023-01-13 14:38:58.893919: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 5
      
      out_dir = output_path / f'clean_averaging_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      ptufile = predict_correct_correlate_ptu(
          files=files_clean2,
          model_id=model_id,
          method='averaging',
          out_path=out_dir)
      
      2023-01-13 14:39:25.593798: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2023-01-13 14:39:25.593871: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2023-01-13 14:39:25.593915: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node117): /proc/driver/nvidia/version does not exist
      2023-01-13 14:39:25.594445: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      a965cd3e-f452-4c79-9428-b5152c3ba8aa
      
    • second: af488luvs and af488 delete correction.
    • #+CALL: prepare-jupyter-averaging() etc…
      /beegfs/ye53nis/drmed-git
      2023-01-16 16:37:55.758832: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2023-01-16 16:37:55.758867: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 5
      
      out_dir = output_path / f'dirty_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      ptufile = predict_correct_correlate_ptu(
          files=files_dirty1,
          model_id=model_id,
          method='delete',
          out_path=out_dir)
      
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      b91bd6bb-e7b8-44c1-b014-02275b3516b6
      
      /beegfs/ye53nis/drmed-git
      2023-01-16 23:53:43.486077: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2023-01-16 23:53:43.486120: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 5
      
      out_dir = output_path / f'dirty_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      ptufile = predict_correct_correlate_ptu(
          files=files_dirty2,
          model_id=model_id,
          method='delete',
          out_path=out_dir)
      
      2023-01-16 23:53:57.181417: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2023-01-16 23:53:57.181479: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2023-01-16 23:53:57.181516: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node117): /proc/driver/nvidia/version does not exist
      2023-01-16 23:53:57.181940: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      /beegfs/ye53nis/drmed-git
      2023-01-17 13:11:45.290135: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2023-01-17 13:11:45.290197: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 5
      
      out_dir = output_path / f'clean_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      ptufile = predict_correct_correlate_ptu(
          files=files_clean1,
          model_id=model_id,
          method='delete',
          out_path=out_dir)
      
      2023-01-17 13:12:04.995182: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2023-01-17 13:12:04.995223: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2023-01-17 13:12:04.995253: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node117): /proc/driver/nvidia/version does not exist
      2023-01-17 13:12:04.995535: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      63efbb75-e4b2-4b2c-ba46-ecf44a9d66fc
      
      /beegfs/ye53nis/drmed-git
      2023-01-17 17:20:22.162939: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2023-01-17 17:20:22.162993: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 5
      
      out_dir = output_path / f'clean_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      ptufile = predict_correct_correlate_ptu(
          files=files_clean2,
          model_id=model_id,
          method='delete',
          out_path=out_dir)
      
      2023-01-17 17:20:51.771891: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
      2023-01-17 17:20:51.771954: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
      2023-01-17 17:20:51.771992: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (node117): /proc/driver/nvidia/version does not exist
      2023-01-17 17:20:51.772457: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
      To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      63efbb75-e4b2-4b2c-ba46-ecf44a9d66fc
      
  • third: tb-pex5 and hs-pex5 with averaging correction and set to zero correction. Because these files are small and memory allocation is less of a problem, we don't need to kill our jupyter session in between
    • #+CALL: prepare-jupyter-averaging("/beegfs/ye53nis/data/191113_Pex5_2_structured")
      /beegfs/ye53nis/drmed-git
      2023-01-18 11:53:14.696643: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
      2023-01-18 11:53:14.696707: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
      
      model_id = 5
      
      out_dir = output_path / f'tbpex5_averaging_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      ptufile = predict_correct_correlate_ptu(
          files=files_dirty,
          model_id=model_id,
          method='averaging',
          out_path=out_dir)
      
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      model_id = 5
      
      out_dir = output_path / f'tbpex5_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      ptufile = predict_correct_correlate_ptu(
          files=files_dirty,
          model_id=model_id,
          method='delete',
          out_path=out_dir)
      
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      model_id = 5
      
      out_dir = output_path / f'hspex5_averaging_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      ptufile = predict_correct_correlate_ptu(
          files=files_clean,
          model_id=model_id,
          method='averaging',
          out_path=out_dir)
      
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      
      model_id = 5
      
      out_dir = output_path / f'hspex5_delete_{model_name_ls[model_id]}/'
      
      os.makedirs(out_dir, exist_ok=True)
      
      ptufile = predict_correct_correlate_ptu(
          files=files_clean,
          model_id=model_id,
          method='delete',
          out_path=out_dir)
      
      WARNING:tensorflow:Unable to restore custom metric. Please ensure that the layer implements `get_config` and `from_config` when saving. In addition, please use the `custom_objects` arg when calling `load_model()`.
      

2.6.13 Analysis 3: correlations and fits of experimental data

2.6.13.1 compare model performance on af488 data
  • call #+CALL: jupyter-set-output-directory()
    ./data/exp-220227-unet/jupyter
    
  • first, illlustrative correlations and fits from FoCuS-point fitting. We start with the data which was later used to plot avg_param data
    • Averaged curves of AF488 (no artifacts) with proceesings (no correction + all models, each curves is averaged from 424 correlations) clean_all_1comp.png clean_all_2comp.png
    • Averaged curves of AF488 + DiO-LUVs (peak artifacts) with proceesings (no correction + all models, each curves is averaged from 440 correlations) dirty_all_1comp.png dirty_all_2comp.png
  • Now we continue with AF488 - this is data without peak artifacts, which was later read in as all_param
    • no correction, 1 species and 2 species fits. The 1 species fit of no correction is the gold standard all correction methods are compared against. clean_no-correction_1comp.png clean_no-correction_2comp.png
    • 0cd20, 1 species and 2 species fits. clean_0cd20_1comp.png clean_0cd20_2comp.png
    • 19e3e, 1 species and 2 species fits. clean_19e3e_1comp.png clean_19e3e_2comp.png
    • 34a6d, 1 species and 2 species fits. clean_34a6d_1comp.png clean_34a6d_2comp.png
    • 484af, 1 species and 2 species fits. clean_484af_1comp.png clean_484af_2comp.png
    • 714af, 1 species and 2 species fits. clean_714af_1comp.png clean_714af_2comp.png
    • 34766, 1 species and 2 species fits. clean_34766_1comp.png clean_34766_2comp.png
    • c1204, 1 species and 2 species fits. clean_c1204_1comp.png clean_c1204_2comp.png
    • fe81d, 1 species and 2 species fits. clean_fe81d_1comp.png clean_fe81d_2comp.png
    • ff67b, 1 species and 2 species fits. clean_ff67b_1comp.png clean_ff67b_2comp.png
  • now illlustrative correlations and fits from AF488 + DiO-LUVs (data with peak artifacts). This is the data we want to correct.
    • no correction, 1 species and 2 species fits. dirty_no-correction_1comp.png dirty_no-correction_2comp.png
    • 0cd20, 1 species and 2 species fits. dirty_0cd20_1comp.png dirty_0cd20_2comp.png
    • 19e3e, 1 species and 2 species fits. dirty_19e3e_1comp.png dirty_19e3e_2comp.png
    • 34a6d, 1 species and 2 species fits. dirty_34a6d_1comp.png dirty_34a6d_2comp.png
    • 484af, 1 species and 2 species fits. dirty_484af_1comp.png dirty_484af_2comp.png
    • 714af, 1 species and 2 species fits. dirty_714af_1comp.png dirty_714af_2comp.png
    • 34766, 1 species and 2 species fits. dirty_34766_1comp.png dirty_34766_2comp.png
    • c1204, 1 species and 2 species fits. dirty_c1204_1comp.png dirty_c1204_2comp.png
    • fe81d, 1 species and 2 species fits. dirty_fe81d_1comp.png dirty_fe81d_2comp.png
    • ff67b, 1 species and 2 species fits. dirty_ff67b_1comp.png dirty_ff67b_2comp.png
  • second, load modules and data
    %cd ~/Programme/drmed-git
    
    /home/lex/Programme/drmed-git
    
    import os
    import numpy as np
    import matplotlib.pyplot as plt
    import pandas as pd
    import seaborn as sns
    from pathlib import Path
    from pprint import pprint
    
    sns.set_theme(style="whitegrid", font_scale=2, palette='colorblind',
                  context='paper')
    
    model_ls = ['ff67be0b68e540a9a29a36a2d0c7a5be', '347669d050f344ad9fb9e480c814f727',
                '714af8cd12c1441eac4ca980e8c20070', '34a6d207ac594035b1009c330fb67a65',
                '484af471c61943fa90e5f78e78a229f0', '0cd2023eeaf745aca0d3e8ad5e1fc653',
                'fe81d71c52404ed790b3a32051258da9', '19e3e786e1bc4e2b93856f5dc9de8216',
                'c1204e3a8a1e4c40a35b5b7b1922d1ce']
    
    model_name_ls = [f'{s:.5}' for s in model_ls]
    
    path = Path('data/exp-220227-unet/2022-05-22_experimental-af488/')
    
    # averaged values
    dirty_avg_1comp_path = path / 'dirty-all-results/dirty_all_1comp_outputParam.csv'
    dirty_avg_2comp_path = path / 'dirty-all-results/dirty_all_2comp_outputParam.csv'
    clean_avg_1comp_path = path / 'clean-all-results/clean_all_1comp_outputParam.csv'
    clean_avg_2comp_path = path / 'clean-all-results/clean_all_2comp_outputParam.csv'
    
    # dirty params
    dirty_noc_1comp_path = path / 'dirty-all-results/dirty_no-correction_1comp_outputParam.csv'
    dirty_noc_2comp_path = path / 'dirty-all-results/dirty_no-correction_2comp_outputParam.csv'
    
    dirty_0cd20_1comp_path = path / 'dirty-all-results/dirty_0cd20_1comp_outputParam.csv'
    dirty_0cd20_2comp_path = path / 'dirty-all-results/dirty_0cd20_2comp_outputParam.csv'
    dirty_19e3e_1comp_path = path / 'dirty-all-results/dirty_19e3e_1comp_outputParam.csv'
    dirty_19e3e_2comp_path = path / 'dirty-all-results/dirty_19e3e_2comp_outputParam.csv'
    dirty_34766_1comp_path = path / 'dirty-all-results/dirty_34766_1comp_outputParam.csv'
    dirty_34766_2comp_path = path / 'dirty-all-results/dirty_34766_2comp_outputParam.csv'
    dirty_34a6d_1comp_path = path / 'dirty-all-results/dirty_34a6d_1comp_outputParam.csv'
    dirty_34a6d_2comp_path = path / 'dirty-all-results/dirty_34a6d_2comp_outputParam.csv'
    dirty_484af_1comp_path = path / 'dirty-all-results/dirty_484af_1comp_outputParam.csv'
    dirty_484af_2comp_path = path / 'dirty-all-results/dirty_484af_2comp_outputParam.csv'
    dirty_714af_1comp_path = path / 'dirty-all-results/dirty_714af_1comp_outputParam.csv'
    dirty_714af_2comp_path = path / 'dirty-all-results/dirty_714af_2comp_outputParam.csv'
    dirty_c1204_1comp_path = path / 'dirty-all-results/dirty_c1204_1comp_outputParam.csv'
    dirty_c1204_2comp_path = path / 'dirty-all-results/dirty_c1204_2comp_outputParam.csv'
    dirty_fe81d_1comp_path = path / 'dirty-all-results/dirty_fe81d_1comp_outputParam.csv'
    dirty_fe81d_2comp_path = path / 'dirty-all-results/dirty_fe81d_2comp_outputParam.csv'
    dirty_ff67b_1comp_path = path / 'dirty-all-results/dirty_ff67b_1comp_outputParam.csv'
    dirty_ff67b_2comp_path = path / 'dirty-all-results/dirty_ff67b_2comp_outputParam.csv'
    
    
    # clean params
    clean_noc_1comp_path = path / 'clean-all-results/clean_no-correction_1comp_outputParam.csv'
    clean_noc_2comp_path = path / 'clean-all-results/clean_no-correction_2comp_outputParam.csv'
    
    clean_0cd20_1comp_path = path / 'clean-all-results/clean_0cd20_1comp_outputParam.csv'
    clean_0cd20_2comp_path = path / 'clean-all-results/clean_0cd20_2comp_outputParam.csv'
    clean_19e3e_1comp_path = path / 'clean-all-results/clean_19e3e_1comp_outputParam.csv'
    clean_19e3e_2comp_path = path / 'clean-all-results/clean_19e3e_2comp_outputParam.csv'
    clean_34766_1comp_path = path / 'clean-all-results/clean_34766_1comp_outputParam.csv'
    clean_34766_2comp_path = path / 'clean-all-results/clean_34766_2comp_outputParam.csv'
    clean_34a6d_1comp_path = path / 'clean-all-results/clean_34a6d_1comp_outputParam.csv'
    clean_34a6d_2comp_path = path / 'clean-all-results/clean_34a6d_2comp_outputParam.csv'
    clean_484af_1comp_path = path / 'clean-all-results/clean_484af_1comp_outputParam.csv'
    clean_484af_2comp_path = path / 'clean-all-results/clean_484af_2comp_outputParam.csv'
    clean_714af_1comp_path = path / 'clean-all-results/clean_714af_1comp_outputParam.csv'
    clean_714af_2comp_path = path / 'clean-all-results/clean_714af_2comp_outputParam.csv'
    clean_c1204_1comp_path = path / 'clean-all-results/clean_c1204_1comp_outputParam.csv'
    clean_c1204_2comp_path = path / 'clean-all-results/clean_c1204_2comp_outputParam.csv'
    clean_fe81d_1comp_path = path / 'clean-all-results/clean_fe81d_1comp_outputParam.csv'
    clean_fe81d_2comp_path = path / 'clean-all-results/clean_fe81d_2comp_outputParam.csv'
    clean_ff67b_1comp_path = path / 'clean-all-results/clean_ff67b_1comp_outputParam.csv'
    clean_ff67b_2comp_path = path / 'clean-all-results/clean_ff67b_2comp_outputParam.csv'
    
    
    # average parameters
    dirty_avg_1comp =  pd.read_csv(dirty_avg_1comp_path, sep=',').assign(
        artifact=10*['af488+luvs',])
    dirty_avg_2comp =  pd.read_csv(dirty_avg_2comp_path, sep=',').assign(
        artifact=10*['af488+luvs',])
    clean_avg_1comp =  pd.read_csv(clean_avg_1comp_path, sep=',').assign(
        artifact=10*['af488',])
    clean_avg_2comp =  pd.read_csv(clean_avg_2comp_path, sep=',').assign(
        artifact=10*['af488',])
    
    # dirty params
    dirty_noc_1comp = pd.read_csv(dirty_noc_1comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['No correction'])
    dirty_noc_2comp = pd.read_csv(dirty_noc_2comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['No correction'])
    
    dirty_0cd20_1comp =  pd.read_csv(dirty_0cd20_1comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['0cd20'])
    dirty_0cd20_2comp =  pd.read_csv(dirty_0cd20_2comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['0cd20'])
    dirty_19e3e_1comp =  pd.read_csv(dirty_19e3e_1comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['19e3e'])
    dirty_19e3e_2comp =  pd.read_csv(dirty_19e3e_2comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['19e3e'])
    dirty_34766_1comp =  pd.read_csv(dirty_34766_1comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['34766'])
    dirty_34766_2comp =  pd.read_csv(dirty_34766_2comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['34766'])
    dirty_34a6d_1comp =  pd.read_csv(dirty_34a6d_1comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['34a6d'])
    dirty_34a6d_2comp =  pd.read_csv(dirty_34a6d_2comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['34a6d'])
    dirty_484af_1comp =  pd.read_csv(dirty_484af_1comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['484af'])
    dirty_484af_2comp =  pd.read_csv(dirty_484af_2comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['484af'])
    dirty_714af_1comp =  pd.read_csv(dirty_714af_1comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['714af'])
    dirty_714af_2comp =  pd.read_csv(dirty_714af_2comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['714af'])
    dirty_c1204_1comp =  pd.read_csv(dirty_c1204_1comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['c1204'])
    dirty_c1204_2comp =  pd.read_csv(dirty_c1204_2comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['c1204'])
    dirty_fe81d_1comp =  pd.read_csv(dirty_fe81d_1comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['fe81d'])
    dirty_fe81d_2comp =  pd.read_csv(dirty_fe81d_2comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['fe81d'])
    dirty_ff67b_1comp =  pd.read_csv(dirty_ff67b_1comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['ff67b'])
    dirty_ff67b_2comp =  pd.read_csv(dirty_ff67b_2comp_path, sep=',').assign(
        artifact=440*['af488+luvs',], processing=440*['ff67b'])
    
    # clean params
    clean_noc_1comp = pd.read_csv(clean_noc_1comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['No correction'])
    clean_noc_2comp = pd.read_csv(clean_noc_2comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['No correction'])
    
    clean_0cd20_1comp =  pd.read_csv(clean_0cd20_1comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['0cd20'])
    clean_0cd20_2comp =  pd.read_csv(clean_0cd20_2comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['0cd20'])
    clean_19e3e_1comp =  pd.read_csv(clean_19e3e_1comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['19e3e'])
    clean_19e3e_2comp =  pd.read_csv(clean_19e3e_2comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['19e3e'])
    clean_34766_1comp =  pd.read_csv(clean_34766_1comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['34766'])
    clean_34766_2comp =  pd.read_csv(clean_34766_2comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['34766'])
    clean_34a6d_1comp =  pd.read_csv(clean_34a6d_1comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['34a6d'])
    clean_34a6d_2comp =  pd.read_csv(clean_34a6d_2comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['34a6d'])
    clean_484af_1comp =  pd.read_csv(clean_484af_1comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['484af'])
    clean_484af_2comp =  pd.read_csv(clean_484af_2comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['484af'])
    clean_714af_1comp =  pd.read_csv(clean_714af_1comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['714af'])
    clean_714af_2comp =  pd.read_csv(clean_714af_2comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['714af'])
    clean_c1204_1comp =  pd.read_csv(clean_c1204_1comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['c1204'])
    clean_c1204_2comp =  pd.read_csv(clean_c1204_2comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['c1204'])
    clean_fe81d_1comp =  pd.read_csv(clean_fe81d_1comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['fe81d'])
    clean_fe81d_2comp =  pd.read_csv(clean_fe81d_2comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['fe81d'])
    clean_ff67b_1comp =  pd.read_csv(clean_ff67b_1comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['ff67b'])
    clean_ff67b_2comp =  pd.read_csv(clean_ff67b_2comp_path, sep=',').assign(
        artifact=424*['af488',], processing=424*['ff67b'])
    
    avg_param = pd.concat([clean_avg_1comp, clean_avg_2comp,
                           dirty_avg_1comp, dirty_avg_2comp])
    
    all_param = pd.concat([clean_noc_1comp, clean_noc_2comp,
                           dirty_noc_1comp, dirty_noc_2comp,
                           dirty_0cd20_1comp, dirty_0cd20_2comp,
                           dirty_19e3e_1comp, dirty_19e3e_2comp,
                           dirty_34766_1comp, dirty_34766_2comp,
                           dirty_34a6d_1comp, dirty_34a6d_2comp,
                           dirty_484af_1comp, dirty_484af_2comp,
                           dirty_714af_1comp, dirty_714af_2comp,
                           dirty_c1204_1comp, dirty_c1204_2comp,
                           dirty_fe81d_1comp, dirty_fe81d_2comp,
                           dirty_ff67b_1comp, dirty_ff67b_2comp,
                           clean_0cd20_1comp, clean_0cd20_2comp,
                           clean_19e3e_1comp, clean_19e3e_2comp,
                           clean_34766_1comp, clean_34766_2comp,
                           clean_34a6d_1comp, clean_34a6d_2comp,
                           clean_484af_1comp, clean_484af_2comp,
                           clean_714af_1comp, clean_714af_2comp,
                           clean_c1204_1comp, clean_c1204_2comp,
                           clean_fe81d_1comp, clean_fe81d_2comp,
                           clean_ff67b_1comp, clean_ff67b_2comp])
    assert set(all_param['Dimen']) == {'3D'}
    assert set(all_param['AR1'] == {5.0})
    assert set(all_param['Diff_eq']) == {'Equation 1B'}
    assert set(all_param['Triplet_eq']) == {'no triplet'}
    assert set(all_param['alpha1']) == {1.0}
    assert set(all_param['xmin']) == {0.001018}
    assert set(all_param['xmax']) == {100.66329, 469.762042}
    assert set(all_param[all_param['xmax'] == 100.66329]['artifact']) == {'af488'}
    assert set(all_param[all_param['xmax'] == 469.762042]['artifact']) == {'af488+luvs'}
    
    avg_param = pd.concat([clean_avg_1comp, clean_avg_2comp,
                           dirty_avg_1comp, dirty_avg_2comp])
    
    assert set(avg_param['Dimen']) == {'3D'}
    assert set(avg_param['AR1'] == {5.0})
    assert set(avg_param['Diff_eq']) == {'Equation 1B'}
    assert set(avg_param['Triplet_eq']) == {'no triplet'}
    assert set(avg_param['alpha1']) == {1.0}
    assert set(avg_param['xmin']) == {0.001018}
    assert set(avg_param['xmax']) == {100.66329, 469.762042}
    assert set(avg_param[avg_param['xmax'] == 100.66329]['artifact']) == {'af488'}
    assert set(avg_param[avg_param['xmax'] == 469.762042]['artifact']) == {'af488+luvs'}
    all_param
    
      nameofplot masterfile parentname parentuqid time of fit Diffeq Diffspecies Tripleteq Tripletspecies Dimen artifact processing A2 stdev(A2) txy2 stdev(txy2) alpha2 stdev(alpha2) AR2 stdev(AR2)
    0 2022-05-26tttr2xfcsCH2BIN1dot020 nM AF4881… Not known tttr2xfcs 0 Mon May 30 14:12:23 2022 Equation 1B 1 no triplet 1 3D af488 No correction NaN NaN NaN NaN NaN NaN NaN NaN
    1 2022-05-26tttr2xfcsCH2BIN1dot020 nM AF4882… Not known tttr2xfcs 0 Mon May 30 14:12:24 2022 Equation 1B 1 no triplet 1 3D af488 No correction NaN NaN NaN NaN NaN NaN NaN NaN
    2 2022-05-26tttr2xfcsCH2BIN1dot020 nM AF4883… Not known tttr2xfcs 0 Mon May 30 14:12:24 2022 Equation 1B 1 no triplet 1 3D af488 No correction NaN NaN NaN NaN NaN NaN NaN NaN
    3 2022-05-26tttr2xfcsCH2BIN1dot020 nM AF4884… Not known tttr2xfcs 0 Mon May 30 14:12:24 2022 Equation 1B 1 no triplet 1 3D af488 No correction NaN NaN NaN NaN NaN NaN NaN NaN
    4 2022-05-26tttr2xfcsCH2BIN1dot020 nM AF4885… Not known tttr2xfcs 0 Mon May 30 14:12:24 2022 Equation 1B 1 no triplet 1 3D af488 No correction NaN NaN NaN NaN NaN NaN NaN NaN
    419 2022-05-25tttr2xfcsCH2BIN1dot020 nM AF4882… Not known tttr2xfcs 0 Mon May 30 14:42:23 2022 Equation 1B 2 no triplet 1 3D af488 ff67b 0.488985 None 0.096094 None 1.0 None 5.0 None
    420 2022-05-25tttr2xfcsCH2BIN1dot020 nM AF4882… Not known tttr2xfcs 0 Mon May 30 14:42:23 2022 Equation 1B 2 no triplet 1 3D af488 ff67b 0.386551 None 0.112745 None 1.0 None 5.0 None
    421 2022-05-25tttr2xfcsCH2BIN1dot020 nM AF4882… Not known tttr2xfcs 0 Mon May 30 14:42:24 2022 Equation 1B 2 no triplet 1 3D af488 ff67b 0.658441 None 0.054065 None 1.0 None 5.0 None
    422 2022-05-25tttr2xfcsCH2BIN1dot020 nM AF4882… Not known tttr2xfcs 0 Mon May 30 14:42:24 2022 Equation 1B 2 no triplet 1 3D af488 ff67b 0.730166 None 0.060976 None 1.0 None 5.0 None
    423 2022-05-25tttr2xfcsCH2BIN1dot020 nM AF4882… Not known tttr2xfcs 0 Mon May 30 14:42:24 2022 Equation 1B 2 no triplet 1 3D af488 ff67b 0.461469 None 0.097456 None 1.0 None 5.0 None

    17280 rows × 39 columns

40 rows × 34 columns

  • first, let's take a look only at avg_param. There, for each model all curves of af488 and af488+luvs were fitted, and then the correlations were averaged. This gives us a good overview with direct comparison of model fit outcome to the fit outcomes without correction. BUT this is not enough to determine that the model is good enough, because in practice, we rarely take 400 times the same measurement and average. So this more resembles the optimal fit outcomes and in the final paper, I analysed the success via fit distributions (so plotting all 400 extracted transit times and comparing the distributions)
    def sort_fit(param_ls):
        nfcs = list(param_ls)[-1]
        array = np.array(list(param_ls)[:-1]).reshape((2, 2))
        # sort by transit times
        array = array[:, array[0, :].argsort()]
        A_fast = float(array[1, 0])
        A_slow = float(array[1, 1])
        N_fast = A_fast * float(nfcs)
        N_slow = A_slow * float(nfcs)
        t_fast = float(array[0, 0]) * 1000
        t_slow = float(array[0, 1]) * 1000
        if np.isnan(t_slow):
            # if tt_low_high[0] <= t_fast <= tt_low_high[1]:
            #     out = f'\\cellcolor[HTML]{{009e73}}${t_fast:.2f}$'
            # else:
            #     out = f'\\cellcolor[HTML]{{d55e00}}${t_fast:.2f}$'
            out = f'$\\tau_D={t_fast:.1f}\\hspace{{1em}}N={nfcs:.1f}$'
        elif f'{A_fast:.0%}' == '100%':
            # if tt_low_high[0] <= t_fast <= tt_low_high[1]:
            #     out = f'\\cellcolor[HTML]{{009e73}}${t_fast:.2f}(100\\%)$'
            # else:
            #     out = f'\\cellcolor[HTML]{{d55e00}}${t_fast:.2f}(100\\%)$'
            out = f'$\\tau_D^{{fast}}={t_fast:.1f}\\hspace{{1em}}N^{{fast}}={N_fast:.1f}$'
        elif f'{A_slow:.0%}' == '100%':
            # if tt_low_high[0] <= t_slow <= tt_low_high[1]:
            #     out = f'\\cellcolor[HTML]{{009e73}}${t_slow:.2f}(100\\%)$'
            # else:
            #     out = f'\\cellcolor[HTML]{{d55e00}}${t_slow:.2f}(100\\%)$'
            out = f'$\\tau_D^{{slow}}={t_slow:.1f}\\hspace{{1em}}N^{{slow}}={N_slow:.1f}$'
        else:
            # if (tt_low_high[0] <= t_fast <= tt_low_high[1]) or (
            #     tt_low_high[0] <= t_slow <= tt_low_high[1]):
            #     out = f'\\cellcolor[HTML]{{009e73}}\\colorbox[HTML]{{009e73}}{{\\makecell{{${t_fast:.2f}^f({A_fast:.0%})$\\\\'\
            #           f'${t_slow:.2f}^s({A_slow:.0%})$}}}}'
            # else:
            #     out = f'\\cellcolor[HTML]{{d55e00}}\\colorbox[HTML]{{d55e00}}{{\\makecell{{${t_fast:.2f}^f({A_fast:.0%})$\\\\'\
            #           f'${t_slow:.2f}^s({A_slow:.0%})$}}}}'
            out = f'\\makecell{{$\\tau_D^{{fast}}={t_fast:.1f}\\hspace{{1em}}N^{{fast}}={N_fast:.1f}$\\\\'\
                  f'$\\tau_D^{{slow}}={t_slow:.1f}\\hspace{{1em}}N^{{slow}}={N_slow:.1f}$}}'
            out = out.replace('%', '\\%')
        return out
    
    avg_param['fit results'] = avg_param[['txy1', 'txy2', 'A1', 'A2', 'N (FCS)']].apply(lambda x: sort_fit(x), axis=1)
    avg_param = avg_param[['name_of_plot', 'Diff_species', 'artifact', 'fit results']]
    avg_param = avg_param.pivot_table(values='fit results',
                                      columns='artifact',
                                      index=['name_of_plot', 'Diff_species'],
                                      aggfunc=lambda x: '-'.join(x))
    avg_param.loc[('clean', 1), 'af488+luvs'] = avg_param.loc[('dirty', 1), 'af488+luvs']
    avg_param.loc[('clean', 2), 'af488+luvs'] = avg_param.loc[('dirty', 2), 'af488+luvs']
    
    avg_param = avg_param.rename(index={'clean' : 'no correction'})
    # to get all models
    first = ['no correction',] + model_name_ls.copy()
    # just two examples
    # first = ['no correction', '0cd20', '34a6d']
    second = [1, 2]
    index_order = pd.MultiIndex.from_product([first, second],
                                             names=[r'\makecell{type of\\processing}', 'fit'])
    avg_param = avg_param.reindex(index=index_order)
    
    with pd.option_context("max_colwidth", 1000):
        print(avg_param.to_latex(escape=False,
                                 column_format='cccc',
                                 caption=(r'Experimental results AF488 data. $\tau_D$ in $\mu s$. For 1 species fit, $N = N(FCS) * T1$. For 2 species fit, $N^{sp} = A^{sp} * (N(FCS) * T1)$ and $\tau_D$ is sorted in fast and slow. If $A^{sp}=100\%$, only display the corresponding $\tau_D$ and $N$ values. 2 species fit results of AF488 in solution has no biophysical relevance and is only shown for completion','Experimental results AF488 data')))
    
    \begin{table}
    \centering
    \caption[Experimental results AF488 data]{Experimental results AF488 data. $\tau_D$ in $\mu s$. For 1 species fit, $N = N(FCS) * T1$. For 2 species fit, $N^{sp} = A^{sp} * (N(FCS) * T1)$ and $\tau_D$ is sorted in fast and slow. If $A^{sp}=100\%$, only display the corresponding $\tau_D$ and $N$ values. 2 species fit results of AF488 in solution has no biophysical relevance and is only shown for completion}
    \begin{tabular}{cccc}
    \toprule
          & artifact &                                                                                                  af488 &                                                                                                 af488+luvs \\
    \makecell{type of\\processing} & fit &                                                                                                        &                                                                                                            \\
    \midrule
    no correction & 1 &                                                                        $\tau_D=39.7\hspace{1em}N=14.3$ &                                                                           $\tau_D=7355.0\hspace{1em}N=4.7$ \\
          & 2 &   \makecell{$\tau_D^{fast}=9.5\hspace{1em}N^{fast}=4.9$\\$\tau_D^{slow}=69.8\hspace{1em}N^{slow}=8.6$} &   \makecell{$\tau_D^{fast}=78.3\hspace{1em}N^{fast}=0.7$\\$\tau_D^{slow}=11944.6\hspace{1em}N^{slow}=3.5$} \\
    ff67b & 1 &                                                                        $\tau_D=39.8\hspace{1em}N=14.3$ &                                                                            $\tau_D=94.0\hspace{1em}N=22.4$ \\
          & 2 &   \makecell{$\tau_D^{fast}=9.9\hspace{1em}N^{fast}=5.0$\\$\tau_D^{slow}=71.7\hspace{1em}N^{slow}=8.4$} &  \makecell{$\tau_D^{fast}=41.8\hspace{1em}N^{fast}=13.1$\\$\tau_D^{slow}=20282.4\hspace{1em}N^{slow}=5.2$} \\
    34766 & 1 &                                                                        $\tau_D=35.1\hspace{1em}N=14.7$ &                                                                            $\tau_D=62.5\hspace{1em}N=23.2$ \\
          & 2 &  \makecell{$\tau_D^{fast}=1.4\hspace{1em}N^{fast}=2.4$\\$\tau_D^{slow}=42.0\hspace{1em}N^{slow}=10.7$} &  \makecell{$\tau_D^{fast}=38.9\hspace{1em}N^{fast}=15.8$\\$\tau_D^{slow}=24434.3\hspace{1em}N^{slow}=4.1$} \\
    714af & 1 &                                                                        $\tau_D=39.8\hspace{1em}N=14.3$ &                                                                           $\tau_D=104.2\hspace{1em}N=22.3$ \\
          & 2 &  \makecell{$\tau_D^{fast}=10.0\hspace{1em}N^{fast}=5.1$\\$\tau_D^{slow}=71.9\hspace{1em}N^{slow}=8.4$} &  \makecell{$\tau_D^{fast}=41.8\hspace{1em}N^{fast}=12.3$\\$\tau_D^{slow}=21435.4\hspace{1em}N^{slow}=5.5$} \\
    34a6d & 1 &                                                                        $\tau_D=39.7\hspace{1em}N=14.3$ &                                                                            $\tau_D=78.9\hspace{1em}N=22.5$ \\
          & 2 &   \makecell{$\tau_D^{fast}=9.4\hspace{1em}N^{fast}=4.8$\\$\tau_D^{slow}=69.6\hspace{1em}N^{slow}=8.6$} &  \makecell{$\tau_D^{fast}=41.9\hspace{1em}N^{fast}=14.2$\\$\tau_D^{slow}=22526.1\hspace{1em}N^{slow}=4.6$} \\
    484af & 1 &                                                                        $\tau_D=39.8\hspace{1em}N=14.3$ &                                                                            $\tau_D=91.2\hspace{1em}N=22.5$ \\
          & 2 &  \makecell{$\tau_D^{fast}=10.1\hspace{1em}N^{fast}=5.1$\\$\tau_D^{slow}=72.2\hspace{1em}N^{slow}=8.4$} &  \makecell{$\tau_D^{fast}=41.2\hspace{1em}N^{fast}=12.8$\\$\tau_D^{slow}=23742.5\hspace{1em}N^{slow}=5.2$} \\
    0cd20 & 1 &                                                                        $\tau_D=39.6\hspace{1em}N=14.3$ &                                                                            $\tau_D=71.3\hspace{1em}N=21.7$ \\
          & 2 &   \makecell{$\tau_D^{fast}=8.9\hspace{1em}N^{fast}=4.6$\\$\tau_D^{slow}=67.5\hspace{1em}N^{slow}=8.8$} &   \makecell{$\tau_D^{fast}=38.8\hspace{1em}N^{fast}=15.5$\\$\tau_D^{slow}=5823.0\hspace{1em}N^{slow}=3.9$} \\
    fe81d & 1 &                                                                        $\tau_D=39.8\hspace{1em}N=14.3$ &                                                                            $\tau_D=90.4\hspace{1em}N=22.3$ \\
          & 2 &  \makecell{$\tau_D^{fast}=10.0\hspace{1em}N^{fast}=5.0$\\$\tau_D^{slow}=71.8\hspace{1em}N^{slow}=8.4$} &  \makecell{$\tau_D^{fast}=42.0\hspace{1em}N^{fast}=13.3$\\$\tau_D^{slow}=19906.1\hspace{1em}N^{slow}=5.0$} \\
    19e3e & 1 &                                                                        $\tau_D=39.6\hspace{1em}N=14.3$ &                                                                            $\tau_D=79.8\hspace{1em}N=22.4$ \\
          & 2 &   \makecell{$\tau_D^{fast}=9.2\hspace{1em}N^{fast}=4.8$\\$\tau_D^{slow}=68.9\hspace{1em}N^{slow}=8.7$} &  \makecell{$\tau_D^{fast}=41.7\hspace{1em}N^{fast}=13.6$\\$\tau_D^{slow}=27489.8\hspace{1em}N^{slow}=4.7$} \\
    c1204 & 1 &                                                                        $\tau_D=29.2\hspace{1em}N=15.2$ &                                                                            $\tau_D=51.4\hspace{1em}N=23.8$ \\
          & 2 &   \makecell{$\tau_D^{fast}=0.1\hspace{1em}N^{fast}=3.9$\\$\tau_D^{slow}=31.4\hspace{1em}N^{slow}=5.0$} &  \makecell{$\tau_D^{fast}=35.5\hspace{1em}N^{fast}=16.4$\\$\tau_D^{slow}=39055.9\hspace{1em}N^{slow}=3.9$} \\
    \bottomrule
    \end{tabular}
    \end{table}
    
  • now I put latex in a simple template of the Overleaf online latex editor like this
    \documentclass{article}
    \usepackage{booktabs} % for \toprule and \midrule in table
    \usepackage{makecell} % for linebreaks in table cells
    
    \begin{document}
    %% here the code abvoe
    \end{document}
    
    
  • the final rendering looks like this:
  • now, we work with all_param. Here, all single correlation curves are loaded without averaging.
    def sort_fit_simple(param_ls):
        nfcs = list(param_ls)[-1]
        array = np.array(list(param_ls)[:-1]).reshape((2, 2))
        # sort by transit times
        array = array[:, array[0, :].argsort()]
        A_fast = float(array[1, 0])
        A_slow = float(array[1, 1])
        N_fast = A_fast * float(nfcs)
        N_slow = A_slow * float(nfcs)
        t_fast = float(array[0, 0])
        t_slow = float(array[0, 1])
        if np.isnan(t_slow):
            # 1-component fit
            out = t_fast, N_fast, pd.NA, pd.NA
        # 2-component fits
        elif f'{A_fast:.0%}' == '100%':
            out = t_fast, N_fast, pd.NA, pd.NA
        elif f'{A_slow:.0%}' == '100%':
            out = t_slow, N_slow, pd.NA, pd.NA
        else:
            out = t_fast, N_fast, t_slow, N_slow
        return out
    
    all_param = all_param.reset_index()
    (all_param['t_fast'], all_param['N_fast'], all_param['t_slow'],
     all_param['N_slow']) = zip(*all_param[['txy1', 'txy2', 'A1', 'A2', 'N (FCS)']].apply(
         lambda x: sort_fit_simple(x), axis=1))
    all_param = all_param[['index', 'Diff_species', 'processing', 'artifact',
                           't_fast', 'N_fast', 't_slow', 'N_slow']]
    all_param = all_param.loc[((all_param['Diff_species'] == 1) & (all_param['artifact'] == 'af488')) |
                              ((all_param['Diff_species'] == 2) & (all_param['artifact'] == 'af488+luvs'))]
    
    N_param = all_param.pivot_table(values='N_fast',
                                    index='index',
                                    columns=['processing', 'artifact', 'Diff_species'])
    
    t_param = all_param.pivot_table(values='t_fast',
                                    index='index',
                                    columns=['processing', 'artifact', 'Diff_species'])
    t_param = t_param * 1000
    print('Statistics of fitted particle numbers')
    display(pd.concat([pd.Series(N_param.median(axis=0), name='median'), N_param.describe().T], axis=1))
    print('Statistics of fitted transit times')
    display(pd.concat([pd.Series(t_param.median(axis=0), name='median'), t_param.describe().T], axis=1))
    
    Statistics of fitted particle numbers
    
          median count mean std min 25% 50% 75% max
    processing artifact Diffspecies                  
    0cd20 af488 1 14.341568 424.0 14.335000 0.263291 13.412392 14.205747 14.341568 14.505898 14.988332
      af488+luvs 2 15.248202 440.0 15.106186 1.484036 6.267150 14.334271 15.248202 16.039287 18.110889
    19e3e af488 1 14.336272 424.0 14.329986 0.263853 13.367317 14.201544 14.336272 14.511038 14.931337
      af488+luvs 2 13.826313 440.0 13.270444 3.181698 0.562121 11.573624 13.826313 15.704750 18.581205
    34766 af488 1 14.702547 424.0 14.684973 0.276662 13.720920 14.551659 14.702547 14.869845 15.347210
      af488+luvs 2 15.732460 440.0 15.267823 2.476381 6.346636 13.704650 15.732460 17.068360 19.961155
    34a6d af488 1 14.335075 424.0 14.325905 0.264438 13.365160 14.194973 14.335075 14.510599 14.933380
      af488+luvs 2 14.302934 440.0 13.641062 2.740812 1.417894 12.297284 14.302934 15.593493 17.885668
    484af af488 1 14.325842 424.0 14.317763 0.263824 13.365161 14.188180 14.325842 14.499389 14.919546
      af488+luvs 2 13.373178 440.0 12.784654 3.023495 1.591405 10.908164 13.373178 15.029049 18.453017
    714af af488 1 14.325842 424.0 14.318594 0.263337 13.365161 14.189044 14.325842 14.497586 14.919546
      af488+luvs 2 12.420635 440.0 12.233933 2.490236 5.295190 10.570730 12.420635 14.066820 17.968677
    No correction af488 1 14.331944 424.0 14.324938 0.264089 13.369703 14.194097 14.331944 14.503913 14.925030
      af488+luvs 2 0.928103 440.0 1.115437 0.832436 0.111298 0.497226 0.928103 1.475898 5.530194
    c1204 af488 1 15.262632 424.0 15.254311 0.321478 13.987100 15.077046 15.262632 15.475146 16.260415
      af488+luvs 2 15.912205 440.0 15.532688 3.448340 1.815959 13.639535 15.912205 18.127480 21.748767
    fe81d af488 1 14.326219 424.0 14.319104 0.263526 13.365161 14.189106 14.326219 14.499389 14.919546
      af488+luvs 2 13.689452 440.0 13.077269 3.152211 0.536799 11.056025 13.689452 15.387298 19.102249
    ff67b af488 1 14.326219 424.0 14.319636 0.263558 13.365161 14.189298 14.326219 14.499389 14.919546
      af488+luvs 2 13.089802 440.0 12.894021 2.446790 4.838165 11.518550 13.089802 14.484227 18.392188
    Statistics of fitted transit times
    
          median count mean std min 25% 50% 75% max
    processing artifact Diffspecies                  
    0cd20 af488 1 39.575746 424.0 39.574374 1.040061 37.190978 38.879728 39.575746 40.279227 42.984704
      af488+luvs 2 38.762520 440.0 38.933931 5.459816 21.356080 35.228893 38.762520 42.262441 62.131261
    19e3e af488 1 39.568778 424.0 39.656947 1.047036 37.102746 38.934719 39.568778 40.416838 42.501109
      af488+luvs 2 41.177209 440.0 41.351740 4.525924 19.027233 38.362287 41.177209 43.854646 60.151240
    34766 af488 1 35.067658 424.0 35.077446 1.109175 32.086412 34.355154 35.067658 35.736697 38.864156
      af488+luvs 2 38.713203 440.0 38.910032 3.349258 27.709990 36.625985 38.713203 41.009242 49.822679
    34a6d af488 1 39.667568 424.0 39.707795 1.038806 36.985999 39.012984 39.667568 40.457557 42.803298
      af488+luvs 2 41.267814 440.0 41.654939 4.707523 30.986888 38.312301 41.267814 44.652582 57.155144
    484af af488 1 39.854056 424.0 39.854861 1.040952 37.153441 39.141726 39.854056 40.582409 42.744448
      af488+luvs 2 40.605601 440.0 41.036124 4.307502 31.922451 38.063892 40.605601 43.505466 63.739400
    714af af488 1 39.835832 424.0 39.838266 1.041757 37.153441 39.107415 39.835832 40.571805 42.696864
      af488+luvs 2 41.430208 440.0 41.791645 4.068318 31.775699 38.784115 41.430208 44.084046 55.334179
    No correction af488 1 39.678008 424.0 39.726486 1.037956 37.049340 39.010144 39.678008 40.444667 42.680513
      af488+luvs 2 59.671435 440.0 295.128650 823.953683 4.884506 42.516272 59.671435 117.828320 9785.473007
    c1204 af488 1 29.177928 424.0 29.247641 1.225078 25.901046 28.399698 29.177928 30.113028 32.740869
      af488+luvs 2 35.278166 440.0 35.503411 2.690742 29.246945 33.580404 35.278166 36.865166 44.796158
    fe81d af488 1 39.830813 424.0 39.828821 1.044278 37.153441 39.089567 39.830813 40.555985 42.744448
      af488+luvs 2 40.439757 440.0 40.766428 6.403163 12.890876 37.417914 40.439757 43.275102 84.641627
    ff67b af488 1 39.803435 424.0 39.830343 1.042279 37.153441 39.087859 39.803435 40.555985 42.744448
      af488+luvs 2 41.451285 440.0 41.757214 4.040506 32.434107 38.893139 41.451285 44.293295 57.788748
  • now we plot fit outcomes (transit times, particle numbers) after neural network prediction and cut and stitch correction. For this AF488 data, we compare the 1 species fit of AF488 in solution with the fast species of the 2 species fit of AF488 + DiO-LUVs. The results show that all models except 34766 and c1204 do quite well in this challenge. Transit times and particle numbers could be restored without introducing false values in clean "AF488 in solution" data.
     pub_param = all_param.replace(['0cd20'], '0cd20: large model\n(200 MB), 6 levels,\npool size=4, scaler\n=quantile transform\n(Gaussian pdf)')
     pub_param = pub_param.replace(['34a6d'], '34a6d: small model\n(7 MB), 3 levels,\npool size=4, scaler=l2')
     pub_param = pub_param.replace(['484af'], '484af: large model\n(275 MB), 7 levels,\npool_size=2, scaler\n=standard')
     pub_param = pub_param.replace(['fe81d'], 'fe81d: large model\n(186 MB), 4 levels,\npool_size=4, scaler\n=standard')
     pub_param = pub_param.replace(['ff67b'], 'ff67b: small model\n(14 MB), 5 levels,\npool_size=4, scaler\n=minmax')
     pub_param = pub_param.replace(['19e3e'], '19e3e: large model\n(172 MB), 3 levels,\npool_size=4, scaler\n=standard')
     pub_param = pub_param.replace(['34766'], '34766: middle-sized\nmodel (73 MB), 5 levels,\npool_size=4, scaler\n=robust')
     pub_param = pub_param.replace(['c1204'], 'c1204: large model\n(312 MB), 9 levels,\npool_size=2, scaler\n=robust')
     pub_param = pub_param.replace(['714af'], '714af: large model\n(234 MB), 5 levels,\npool_size=4, scaler\n=maxabs')
     g = sns.catplot(data=pub_param,
                     y='t_fast',
                     x='processing',
                     hue='artifact',
                     sharey=True,
                     height=10,
                     aspect=3.3,
                     legend_out=True,
                     kind='boxen',
                     showfliers=False)
     g._legend.set_title('')
    
     g._legend.remove()
     for i, ax in enumerate(g.axes):
         clean = pub_param[(pub_param['processing'] == 'No correction') &
                           (pub_param['artifact'] == 'af488')]
         median = clean['t_fast'].median()
         line = ax[0].axhline(median, lw=4, label='', ls='--')
         line_legend = {f'\n$\\tau_{{exp}}={median:.2f}ms$' : line}
         g._legend_data.update(line_legend)
     g.add_legend(g._legend_data)
     new_labels = ['424 traces of\nAlexaFluor488\n(no artifacts)\n$\\tau_D$ from\n1 species fit',
                   '440 traces of\nAF488 + DiO LUVs\n(peak artifacts)\n$\\tau_D$ from\nfast sp. of 2 sp. fit']
     for t, l in zip(g._legend.texts, new_labels):
         t.set_text(l)
    
     g.map_dataframe(sns.stripplot,
           y='t_fast',
           x='processing',
           hue='artifact',
           dodge=True,
           palette=sns.color_palette(['0.3']),
           size=4,
           jitter=0.2)
     g.fig.suptitle('Simulation → prediction → correction pipeline successfully restores transit times',
                    size=25)
    plt.setp(g.axes, yscale='log', xlabel='',
              ylabel=r'log transit time $\tau_{D}$ $[ms]$')
    
     g.tight_layout()
    
     savefig = f'./data/exp-220227-unet/jupyter/analysis3_af488_compare-transit-times'
     plt.savefig(f'{savefig}.pdf', dpi=300)
     # os.system(f'pdf2svg {savefig}.pdf {savefig}.svg')
     plt.close('all')
    
     g = sns.catplot(data=pub_param,
                     y='N_fast',
                     x='processing',
                     hue='artifact',
                     sharey=True,
                     height=10.3,
                     aspect=3.3,
                     legend_out=True,
                     kind='boxen',
                     showfliers=False)
     g._legend.set_title('')
     g._legend.remove()
     for i, ax in enumerate(g.axes):
         clean = pub_param[(pub_param['processing'] == 'No correction') &
                           (pub_param['artifact'] == 'af488')]
         median = clean['N_fast'].median()
         line = ax[0].axhline(median, lw=4, label='', ls=':')
         line_legend = {f'\n$N{{exp}}={median:.2f}$' : line}
         g._legend_data.update(line_legend)
     g.add_legend(g._legend_data)
     new_labels = ['424 traces of\nAlexaFluor488\n(no artifacts)\n$N (FCS)$ from\n1 species fit',
                   '440 traces of\nAF488 + DiO LUVs\n(peak artifacts)\n$N (FCS)\\cdot A$ from\nfast sp. of 2 sp. fit']
     for t, l in zip(g._legend.texts, new_labels):
         t.set_text(l)
    
     g.map_dataframe(sns.stripplot,
           y='N_fast',
           x='processing',
           hue='artifact',
           dodge=True,
           palette=sns.color_palette(['0.3']),
           size=4,
           jitter=0.2)
     g.fig.suptitle('Simulation → prediction → correction pipeline successfully restores particle number',
                    size=25)
    
     plt.setp(g.axes, xlabel='',
              ylabel=r'particle number $N$ $[fl^{-1}]$')
    
     g.tight_layout()
     savefig = f'./data/exp-220227-unet/jupyter/analysis3_af488_compare-particle-numbers'
     plt.savefig(f'{savefig}.pdf', dpi=300)
     # os.system(f'pdf2svg {savefig}.pdf {savefig}.svg')
     plt.close('all')
    
  • this is the two plots as PDFs, first the comparison of transit times, second the comparison of particle numbers
2.6.13.2 compare model performance on pex5 data
  • call #+CALL: jupyter-set-output-directory()
    ./data/exp-220227-unet/jupyter
    
  • first, illlustrative correlations and fits from FoCuS-point fitting. We start with the data which was later used to plot avg_param data
    • Averaged curves of Hs-PEX5-eGFP (no artifacts) with proceesings (no correction + all models, each curves is averaged from 250 correlations) Hs-PEX5-eGFP_avg_cas_1comp.png Hs-PEX5-eGFP_avg_cas_2comp.png
    • Averaged curves of Tb-PEX5-eGFP (peak artifacts) with proceesings (no correction + all models, each curves is averaged from 250 correlations) Tb-PEX5-eGFP_avg_cas_1comp.png Tb-PEX5-eGFP_avg_cas_2comp.png
  • Now we continue with Hs-PEX5-eGFP - this is data without peak artifacts, which was later read in as all_param
    • no correction, 1 species and 2 species fits. The 1 species fit of no correction is the gold standard all correction methods are compared against. Hs-PEX5-eGFP_no-correction_1comp.png Hs-PEX5-eGFP_no-correction_2comp.png
      • As can be noticed, the amplitudes vary widely. There probably was an instability in the measurement. Here the curves 1-25 Hs-PEX5-eGFP_no-correction-curves1-25_1comp.png
      • curves 26-50 Hs-PEX5-eGFP_no-correction-curves26-50_1comp.png
      • curves 51-75 Hs-PEX5-eGFP_no-correction-curves51-75_1comp.png
      • curves 76-100 Hs-PEX5-eGFP_no-correction-curves76-100_1comp.png
      • curves 101-125 Hs-PEX5-eGFP_no-correction-curves101-125_1comp.png
      • curves 126-150 Hs-PEX5-eGFP_no-correction-curves126-150_1comp.png
      • curves 151-175 Hs-PEX5-eGFP_no-correction-curves151-175_1comp.png
      • curves 176-200 Hs-PEX5-eGFP_no-correction-curves176-200_1comp.png
      • curves 201-225 Hs-PEX5-eGFP_no-correction-curves201-225_1comp.png
      • curves 226-250 Hs-PEX5-eGFP_no-correction-curves226-250_1comp.png
    • 0cd20, 1 species and 2 species fits. Hs-PEX5-eGFP_0cd20_1comp.png Hs-PEX5-eGFP_0cd20_2comp.png
    • 19e3e, 1 species and 2 species fits. Hs-PEX5-eGFP_19e3e_1comp.png Hs-PEX5-eGFP_19e3e_2comp.png
    • 34a6d, 1 species and 2 species fits. Hs-PEX5-eGFP_34a6d_1comp.png Hs-PEX5-eGFP_34a6d_2comp.png
    • 484af, 1 species and 2 species fits. Hs-PEX5-eGFP_484af_1comp.png Hs-PEX5-eGFP_484af_2comp.png
    • 714af, 1 species and 2 species fits. Hs-PEX5-eGFP_714af_1comp.png Hs-PEX5-eGFP_714af_2comp.png
    • 34766, 1 species and 2 species fits. Hs-PEX5-eGFP_34766_1comp.png Hs-PEX5-eGFP_34766_2comp.png
    • c1204, 1 species and 2 species fits. Hs-PEX5-eGFP_c1204_1comp.png Hs-PEX5-eGFP_c1204_2comp.png
    • fe81d, 1 species and 2 species fits. Hs-PEX5-eGFP_fe81d_1comp.png Hs-PEX5-eGFP_fe81d_2comp.png
    • ff67b, 1 species and 2 species fits. Hs-PEX5-eGFP_ff67b_1comp.png Hs-PEX5-eGFP_ff67b_2comp.png
  • now illlustrative correlations and fits from AF488 + DiO-LUVs (data with peak artifacts). This is the data we want to correct.
    • no correction, 1 species and 2 species fits. Tb-PEX5-eGFP_no-correction_1comp.png Tb-PEX5-eGFP_no-correction_2comp.png
    • 0cd20, 1 species and 2 species fits. Tb-PEX5-eGFP_0cd20_1comp.png Tb-PEX5-eGFP_0cd20_2comp.png
    • 19e3e, 1 species and 2 species fits. Tb-PEX5-eGFP_19e3e_1comp.png Tb-PEX5-eGFP_19e3e_2comp.png
    • 34a6d, 1 species and 2 species fits. Tb-PEX5-eGFP_34a6d_1comp.png Tb-PEX5-eGFP_34a6d_2comp.png
    • 484af, 1 species and 2 species fits. Tb-PEX5-eGFP_484af_1comp.png Tb-PEX5-eGFP_484af_2comp.png
    • 714af, 1 species and 2 species fits. Tb-PEX5-eGFP_714af_1comp.png Tb-PEX5-eGFP_714af_2comp.png
    • 34766, 1 species and 2 species fits. Tb-PEX5-eGFP_34766_1comp.png Tb-PEX5-eGFP_34766_2comp.png
    • c1204, 1 species and 2 species fits. Tb-PEX5-eGFP_c1204_1comp.png Tb-PEX5-eGFP_c1204_2comp.png
    • fe81d, 1 species and 2 species fits. Tb-PEX5-eGFP_fe81d_1comp.png Tb-PEX5-eGFP_fe81d_2comp.png
    • ff67b, 1 species and 2 species fits. Tb-PEX5-eGFP_ff67b_1comp.png Tb-PEX5-eGFP_ff67b_2comp.png
  • second, load modules and data
    %cd ~/Programme/drmed-git
    
    import os
    import numpy as np
    import matplotlib.pyplot as plt
    import pandas as pd
    import seaborn as sns
    from pathlib import Path
    from pprint import pprint
    
    sns.set_theme(style="whitegrid", font_scale=2.5, palette='colorblind',
                  context='paper')
    
    model_ls = ['ff67be0b68e540a9a29a36a2d0c7a5be', '347669d050f344ad9fb9e480c814f727',
                '714af8cd12c1441eac4ca980e8c20070', '34a6d207ac594035b1009c330fb67a65',
                '484af471c61943fa90e5f78e78a229f0', '0cd2023eeaf745aca0d3e8ad5e1fc653',
                'fe81d71c52404ed790b3a32051258da9', '19e3e786e1bc4e2b93856f5dc9de8216',
                'c1204e3a8a1e4c40a35b5b7b1922d1ce']
    
    model_name_ls = [f'{s:.5}' for s in model_ls]
    
    path = Path('data/exp-220227-unet/2022-06-02_experimental-pex5/')
    
    # averaged values
    dirty_avg_1comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_avg_cas_1comp_outputParam.csv'
    dirty_avg_2comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_avg_cas_2comp_outputParam.csv'
    clean_avg_1comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_avg_cas_1comp_outputParam.csv'
    clean_avg_2comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_avg_cas_2comp_outputParam.csv'
    
    # dirty params
    dirty_noc_1comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_no-correction_1comp_outputParam.csv'
    dirty_noc_2comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_no-correction_2comp_outputParam.csv'
    
    dirty_0cd20_1comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_0cd20_1comp_outputParam.csv'
    dirty_0cd20_2comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_0cd20_2comp_outputParam.csv'
    dirty_19e3e_1comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_19e3e_1comp_outputParam.csv'
    dirty_19e3e_2comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_19e3e_2comp_outputParam.csv'
    dirty_34766_1comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_34766_1comp_outputParam.csv'
    dirty_34766_2comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_34766_2comp_outputParam.csv'
    dirty_34a6d_1comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_34a6d_1comp_outputParam.csv'
    dirty_34a6d_2comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_34a6d_2comp_outputParam.csv'
    dirty_484af_1comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_484af_1comp_outputParam.csv'
    dirty_484af_2comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_484af_2comp_outputParam.csv'
    dirty_714af_1comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_714af_1comp_outputParam.csv'
    dirty_714af_2comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_714af_2comp_outputParam.csv'
    dirty_c1204_1comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_c1204_1comp_outputParam.csv'
    dirty_c1204_2comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_c1204_2comp_outputParam.csv'
    dirty_fe81d_1comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_fe81d_1comp_outputParam.csv'
    dirty_fe81d_2comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_fe81d_2comp_outputParam.csv'
    dirty_ff67b_1comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_ff67b_1comp_outputParam.csv'
    dirty_ff67b_2comp_path = path / 'dirty-all-results/Tb-PEX5-eGFP_ff67b_2comp_outputParam.csv'
    
    # clean params
    clean_noc_1comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_no-correction_1comp_outputParam.csv'
    clean_noc_2comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_no-correction_2comp_outputParam.csv'
    
    clean_0cd20_1comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_0cd20_1comp_outputParam.csv'
    clean_0cd20_2comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_0cd20_2comp_outputParam.csv'
    clean_19e3e_1comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_19e3e_1comp_outputParam.csv'
    clean_19e3e_2comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_19e3e_2comp_outputParam.csv'
    clean_34766_1comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_34766_1comp_outputParam.csv'
    clean_34766_2comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_34766_2comp_outputParam.csv'
    clean_34a6d_1comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_34a6d_1comp_outputParam.csv'
    clean_34a6d_2comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_34a6d_2comp_outputParam.csv'
    clean_484af_1comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_484af_1comp_outputParam.csv'
    clean_484af_2comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_484af_2comp_outputParam.csv'
    clean_714af_1comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_714af_1comp_outputParam.csv'
    clean_714af_2comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_714af_2comp_outputParam.csv'
    clean_c1204_1comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_c1204_1comp_outputParam.csv'
    clean_c1204_2comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_c1204_2comp_outputParam.csv'
    clean_fe81d_1comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_fe81d_1comp_outputParam.csv'
    clean_fe81d_2comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_fe81d_2comp_outputParam.csv'
    clean_ff67b_1comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_ff67b_1comp_outputParam.csv'
    clean_ff67b_2comp_path = path / 'clean-all-results/Hs-PEX5-eGFP_ff67b_2comp_outputParam.csv'
    
    # average parameters
    dirty_avg_1comp =  pd.read_csv(dirty_avg_1comp_path, sep=',').assign(
        artifact=10*['Tb-PEX5-eGFP',])
    dirty_avg_2comp =  pd.read_csv(dirty_avg_2comp_path, sep=',').assign(
        artifact=10*['Tb-PEX5-eGFP',])
    clean_avg_1comp =  pd.read_csv(clean_avg_1comp_path, sep=',').assign(
        artifact=10*['Hs-PEX5-eGFP',])
    clean_avg_2comp =  pd.read_csv(clean_avg_2comp_path, sep=',').assign(
        artifact=10*['Hs-PEX5-eGFP',])
    
    # dirty params
    dirty_noc_1comp = pd.read_csv(dirty_noc_1comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['No correction'])
    dirty_noc_2comp = pd.read_csv(dirty_noc_2comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['No correction'])
    
    dirty_0cd20_1comp =  pd.read_csv(dirty_0cd20_1comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['0cd20'])
    dirty_0cd20_2comp =  pd.read_csv(dirty_0cd20_2comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['0cd20'])
    dirty_19e3e_1comp =  pd.read_csv(dirty_19e3e_1comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['19e3e'])
    dirty_19e3e_2comp =  pd.read_csv(dirty_19e3e_2comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['19e3e'])
    dirty_34766_1comp =  pd.read_csv(dirty_34766_1comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['34766'])
    dirty_34766_2comp =  pd.read_csv(dirty_34766_2comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['34766'])
    dirty_34a6d_1comp =  pd.read_csv(dirty_34a6d_1comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['34a6d'])
    dirty_34a6d_2comp =  pd.read_csv(dirty_34a6d_2comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['34a6d'])
    dirty_484af_1comp =  pd.read_csv(dirty_484af_1comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['484af'])
    dirty_484af_2comp =  pd.read_csv(dirty_484af_2comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['484af'])
    dirty_714af_1comp =  pd.read_csv(dirty_714af_1comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['714af'])
    dirty_714af_2comp =  pd.read_csv(dirty_714af_2comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['714af'])
    dirty_c1204_1comp =  pd.read_csv(dirty_c1204_1comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['c1204'])
    dirty_c1204_2comp =  pd.read_csv(dirty_c1204_2comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['c1204'])
    dirty_fe81d_1comp =  pd.read_csv(dirty_fe81d_1comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['fe81d'])
    dirty_fe81d_2comp =  pd.read_csv(dirty_fe81d_2comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['fe81d'])
    dirty_ff67b_1comp =  pd.read_csv(dirty_ff67b_1comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['ff67b'])
    dirty_ff67b_2comp =  pd.read_csv(dirty_ff67b_2comp_path, sep=',').assign(
        artifact=250*['Tb-PEX5-eGFP',], processing=250*['ff67b'])
    
    # clean params
    clean_noc_1comp = pd.read_csv(clean_noc_1comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['No correction'])
    clean_noc_2comp = pd.read_csv(clean_noc_2comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['No correction'])
    
    clean_0cd20_1comp =  pd.read_csv(clean_0cd20_1comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['0cd20'])
    clean_0cd20_2comp =  pd.read_csv(clean_0cd20_2comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['0cd20'])
    clean_19e3e_1comp =  pd.read_csv(clean_19e3e_1comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['19e3e'])
    clean_19e3e_2comp =  pd.read_csv(clean_19e3e_2comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['19e3e'])
    clean_34766_1comp =  pd.read_csv(clean_34766_1comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['34766'])
    clean_34766_2comp =  pd.read_csv(clean_34766_2comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['34766'])
    clean_34a6d_1comp =  pd.read_csv(clean_34a6d_1comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['34a6d'])
    clean_34a6d_2comp =  pd.read_csv(clean_34a6d_2comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['34a6d'])
    clean_484af_1comp =  pd.read_csv(clean_484af_1comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['484af'])
    clean_484af_2comp =  pd.read_csv(clean_484af_2comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['484af'])
    clean_714af_1comp =  pd.read_csv(clean_714af_1comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['714af'])
    clean_714af_2comp =  pd.read_csv(clean_714af_2comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['714af'])
    clean_c1204_1comp =  pd.read_csv(clean_c1204_1comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['c1204'])
    clean_c1204_2comp =  pd.read_csv(clean_c1204_2comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['c1204'])
    clean_fe81d_1comp =  pd.read_csv(clean_fe81d_1comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['fe81d'])
    clean_fe81d_2comp =  pd.read_csv(clean_fe81d_2comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['fe81d'])
    clean_ff67b_1comp =  pd.read_csv(clean_ff67b_1comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['ff67b'])
    clean_ff67b_2comp =  pd.read_csv(clean_ff67b_2comp_path, sep=',').assign(
        artifact=250*['Hs-PEX5-eGFP',], processing=250*['ff67b'])
    
    avg_param = pd.concat([clean_avg_1comp, clean_avg_2comp,
                           dirty_avg_1comp, dirty_avg_2comp])
    assert set(avg_param['Dimen']) == {'3D'}
    assert set(avg_param['AR1'] == {6.0})
    assert set(avg_param['Diff_eq']) == {'Equation 1B'}
    assert set(avg_param['Triplet_eq']) == {'Triplet Eq 2B'}
    assert set(avg_param['tauT1']) == {0.04}
    assert set(avg_param['alpha1']) == {1.0}
    assert set(avg_param['xmin']) == {0.001018}
    assert set(avg_param['xmax']) == {939.52409}
    
    all_param = pd.concat([clean_noc_1comp, clean_noc_2comp,
                           dirty_noc_1comp, dirty_noc_2comp,
                           dirty_0cd20_1comp, dirty_0cd20_2comp,
                           dirty_19e3e_1comp, dirty_19e3e_2comp,
                           dirty_34766_1comp, dirty_34766_2comp,
                           dirty_34a6d_1comp, dirty_34a6d_2comp,
                           dirty_484af_1comp, dirty_484af_2comp,
                           dirty_714af_1comp, dirty_714af_2comp,
                           dirty_c1204_1comp, dirty_c1204_2comp,
                           dirty_fe81d_1comp, dirty_fe81d_2comp,
                           dirty_ff67b_1comp, dirty_ff67b_2comp,
                           clean_0cd20_1comp, clean_0cd20_2comp,
                           clean_19e3e_1comp, clean_19e3e_2comp,
                           clean_34766_1comp, clean_34766_2comp,
                           clean_34a6d_1comp, clean_34a6d_2comp,
                           clean_484af_1comp, clean_484af_2comp,
                           clean_714af_1comp, clean_714af_2comp,
                           clean_c1204_1comp, clean_c1204_2comp,
                           clean_fe81d_1comp, clean_fe81d_2comp,
                           clean_ff67b_1comp, clean_ff67b_2comp])
    assert set(all_param['Dimen']) == {'3D'}
    assert set(all_param['AR1'] == {6.0})
    assert set(all_param['Diff_eq']) == {'Equation 1B'}
    assert set(all_param['Triplet_eq']) == {'Triplet Eq 2B'}
    assert set(all_param['tauT1']) == {0.04}
    assert set(all_param['alpha1']) == {1.0}
    assert set(all_param['xmin']) == {0.001018}
    assert set(all_param['xmax']) == {939.52409}
    
    all_param
    
      nameofplot masterfile parentname parentuqid time of fit Diffeq Diffspecies Tripleteq Tripletspecies Dimen artifact processing A2 stdev(A2) txy2 stdev(txy2) alpha2 stdev(alpha2) AR2 stdev(AR2)
    0 2022-06-02tttr2xfcsCH2BIN1dot0HsPEX5EGFP 1… Not known tttr2xfcs 0 Mon Aug 1 13:38:08 2022 Equation 1B 1 Triplet Eq 2B 1 3D Hs-PEX5-eGFP No correction NaN NaN NaN NaN NaN NaN NaN NaN
    1 2022-06-02tttr2xfcsCH2BIN1dot0HsPEX5EGFP 1… Not known tttr2xfcs 0 Mon Aug 1 13:38:08 2022 Equation 1B 1 Triplet Eq 2B 1 3D Hs-PEX5-eGFP No correction NaN NaN NaN NaN NaN NaN NaN NaN
    2 2022-06-02tttr2xfcsCH2BIN1dot0HsPEX5EGFP 1… Not known tttr2xfcs 0 Mon Aug 1 13:38:08 2022 Equation 1B 1 Triplet Eq 2B 1 3D Hs-PEX5-eGFP No correction NaN NaN NaN NaN NaN NaN NaN NaN
    3 2022-06-02tttr2xfcsCH2BIN1dot0HsPEX5EGFP 1… Not known tttr2xfcs 0 Mon Aug 1 13:38:08 2022 Equation 1B 1 Triplet Eq 2B 1 3D Hs-PEX5-eGFP No correction NaN NaN NaN NaN NaN NaN NaN NaN
    4 2022-06-02tttr2xfcsCH2BIN1dot0HsPEX5EGFP 1… Not known tttr2xfcs 0 Mon Aug 1 13:38:08 2022 Equation 1B 1 Triplet Eq 2B 1 3D Hs-PEX5-eGFP No correction NaN NaN NaN NaN NaN NaN NaN NaN
    245 2022-06-03tttr2xfcsCH2BIN1dot0HsPEX5EGFP 1… Not known tttr2xfcs 0 Mon Aug 1 13:51:37 2022 Equation 1B 2 Triplet Eq 2B 1 3D Hs-PEX5-eGFP ff67b 0.294062 None 0.221001 None 1.0 None 6.0 None
    246 2022-06-03tttr2xfcsCH2BIN1dot0HsPEX5EGFP 1… Not known tttr2xfcs 0 Mon Aug 1 13:51:37 2022 Equation 1B 2 Triplet Eq 2B 1 3D Hs-PEX5-eGFP ff67b 0.334689 None 0.203118 None 1.0 None 6.0 None
    247 2022-06-03tttr2xfcsCH2BIN1dot0HsPEX5EGFP 1… Not known tttr2xfcs 0 Mon Aug 1 13:51:37 2022 Equation 1B 2 Triplet Eq 2B 1 3D Hs-PEX5-eGFP ff67b 0.285890 None 0.198634 None 1.0 None 6.0 None
    248 2022-06-03tttr2xfcsCH2BIN1dot0HsPEX5EGFP 1… Not known tttr2xfcs 0 Mon Aug 1 13:51:37 2022 Equation 1B 2 Triplet Eq 2B 1 3D Hs-PEX5-eGFP ff67b 0.334689 None 0.203118 None 1.0 None 6.0 None
    249 2022-06-03tttr2xfcsCH2BIN1dot0HsPEX5EGFP 1… Not known tttr2xfcs 0 Mon Aug 1 13:51:37 2022 Equation 1B 2 Triplet Eq 2B 1 3D Hs-PEX5-eGFP ff67b 0.334689 None 0.203118 None 1.0 None 6.0 None

    10000 rows × 43 columns

  • first, let's again take a look only at avg_param. There, for each model all curves of hs-pex5-egfp and tb-pex5-egfp were fitted, and then the correlations were averaged. This gives us a good overview with direct comparison of model fit outcome to the fit outcomes without correction. BUT this is not enough to determine that the model is good enough, because in practice, we rarely take 250 times the same measurement and average. So this more resembles the optimal fit outcomes and in the final paper, I analysed the success via fit distributions (so plotting all 250 extracted transit times and comparing the distributions, see later)
    def sort_fit(param_ls):
        nfcs = list(param_ls)[-1]
        triplet = list(param_ls)[-2]
        array = np.array(list(param_ls)[:-2]).reshape((2, 2))
        # sort by transit times
        array = array[:, array[0, :].argsort()]
        A_fast = float(array[1, 0])
        A_slow = float(array[1, 1])
        N_fast = A_fast * (triplet * float(nfcs))
        N_slow = A_slow * (triplet * float(nfcs))
        t_fast = float(array[0, 0])
        t_slow = float(array[0, 1])
        if np.isnan(t_slow):
            # if tt_low_high[0] <= t_fast <= tt_low_high[1]:
            #     out = f'\\cellcolor[HTML]{{009e73}}${t_fast:.2f}$'
            # else:
            #     out = f'\\cellcolor[HTML]{{d55e00}}${t_fast:.2f}$'
            out = f'$\\tau_D={t_fast:.2f}\\hspace{{3em}}N={nfcs:.1f}$'
        elif f'{A_fast:.0%}' == '100%':
            # if tt_low_high[0] <= t_fast <= tt_low_high[1]:
            #     out = f'\\cellcolor[HTML]{{009e73}}${t_fast:.2f}(100\\%)$'
            # else:
            #     out = f'\\cellcolor[HTML]{{d55e00}}${t_fast:.2f}(100\\%)$'
            out = f'$\\tau_D^{{fast}}={t_fast:.2f}\\hspace{{1em}}N^{{fast}}={N_fast:.1f}$'
        elif f'{A_slow:.0%}' == '100%':
            # if tt_low_high[0] <= t_slow <= tt_low_high[1]:
            #     out = f'\\cellcolor[HTML]{{009e73}}${t_slow:.2f}(100\\%)$'
            # else:
            #     out = f'\\cellcolor[HTML]{{d55e00}}${t_slow:.2f}(100\\%)$'
            out = f'$\\tau_D^{{slow}}={t_slow:.2f}\\hspace{{1em}}N^{{slow}}={N_slow:.1f}$'
        else:
            # if (tt_low_high[0] <= t_fast <= tt_low_high[1]) or (
            #     tt_low_high[0] <= t_slow <= tt_low_high[1]):
            #     out = f'\\cellcolor[HTML]{{009e73}}\\colorbox[HTML]{{009e73}}{{\\makecell{{${t_fast:.2f}^f({A_fast:.0%})$\\\\'\
            #           f'${t_slow:.2f}^s({A_slow:.0%})$}}}}'
            # else:
            #     out = f'\\cellcolor[HTML]{{d55e00}}\\colorbox[HTML]{{d55e00}}{{\\makecell{{${t_fast:.2f}^f({A_fast:.0%})$\\\\'\
            #           f'${t_slow:.2f}^s({A_slow:.0%})$}}}}'
            out = f'\\makecell{{$\\tau_D^{{fast}}={t_fast:.2f}\\hspace{{1em}}N^{{fast}}={N_fast:.1f}$\\\\'\
                  f'$\\tau_D^{{slow}}={t_slow:.2f}\\hspace{{1em}}N^{{slow}}={N_slow:.1f}$}}'
            out = out.replace('%', '\\%')
        return out
    
    avg_param['fit results'] = avg_param[['txy1', 'txy2', 'A1', 'A2', 'T1', 'N (FCS)']].apply(lambda x: sort_fit(x), axis=1)
    avg_param = avg_param[['name_of_plot', 'Diff_species', 'artifact', 'fit results']]
    avg_param = avg_param.pivot_table(values='fit results',
                                      columns='artifact',
                                      index=['name_of_plot', 'Diff_species'],
                                      aggfunc=lambda x: '-'.join(x))
    avg_param.loc[('clean', 1), 'Tb-PEX5-eGFP'] = avg_param.loc[('dirty', 1), 'Tb-PEX5-eGFP']
    avg_param.loc[('clean', 2), 'Tb-PEX5-eGFP'] = avg_param.loc[('dirty', 2), 'Tb-PEX5-eGFP']
    
    avg_param = avg_param.rename(index={'clean' : 'no correction'})
    # to get all models
    first = ['no correction',] + model_name_ls.copy()
    # just two examples
    # first = ['no correction', '0cd20', '34a6d']
    second = [1, 2]
    index_order = pd.MultiIndex.from_product([first, second],
                                             names=[r'\makecell{type of\\processing}', 'fit'])
    avg_param = avg_param.reindex(index=index_order)
    
    with pd.option_context("max_colwidth", 1000):
        print(avg_param.to_latex(escape=False,
                                 column_format='ccll',
                                 caption=(r'Experimental results PEX5 data. $\tau_D$ in $ms$. For 1 species fit, $N = N(FCS)$. For 2 species fit, $N^{sp} = A^{sp} * N(FCS)$ and $\tau_D$ is sorted in fast and slow. If $A^{sp}=100\%$, only display the corresponding $\tau_D$ and $N$ values','Experimental results PEX5 data.')))
    
    \begin{table}
    \centering
    \caption[Experimental results PEX5 data.]{Experimental results PEX5 data. $\tau_D$ in $ms$. For 1 species fit, $N = N(FCS)$. For 2 species fit, $N^{sp} = A^{sp} * N(FCS)$ and $\tau_D$ is sorted in fast and slow. If $A^{sp}=100\%$, only display the corresponding $\tau_D$ and $N$ values}
    \begin{tabular}{ccll}
    \toprule
          & artifact &                                                                                              Hs-PEX5-eGFP &                                                                                             Tb-PEX5-eGFP \\
    \makecell{type of\\processing} & fit &                                                                                                           &                                                                                                          \\
    \midrule
    no correction & 1 &                                                                            $\tau_D=0.36\hspace{3em}N=1.1$ &                                                                           $\tau_D=0.52\hspace{3em}N=0.8$ \\
          & 2 &     \makecell{$\tau_D^{fast}=0.36\hspace{1em}N^{fast}=0.1$\\$\tau_D^{slow}=0.36\hspace{1em}N^{slow}=0.1$} &   \makecell{$\tau_D^{fast}=0.45\hspace{1em}N^{fast}=0.2$\\$\tau_D^{slow}=13.44\hspace{1em}N^{slow}=0.0$} \\
    ff67b & 1 &                                                                            $\tau_D=0.23\hspace{3em}N=2.1$ &                                                                           $\tau_D=0.28\hspace{3em}N=1.5$ \\
          & 2 &    \makecell{$\tau_D^{fast}=0.21\hspace{1em}N^{fast}=0.5$\\$\tau_D^{slow}=86.83\hspace{1em}N^{slow}=0.0$} &   \makecell{$\tau_D^{fast}=0.24\hspace{1em}N^{fast}=0.4$\\$\tau_D^{slow}=31.23\hspace{1em}N^{slow}=0.0$} \\
    34766 & 1 &                                                                            $\tau_D=0.39\hspace{3em}N=3.6$ &                                                                           $\tau_D=1.12\hspace{3em}N=4.1$ \\
          & 2 &     \makecell{$\tau_D^{fast}=0.05\hspace{1em}N^{fast}=0.0$\\$\tau_D^{slow}=6.32\hspace{1em}N^{slow}=0.0$} &    \makecell{$\tau_D^{fast}=0.04\hspace{1em}N^{fast}=0.0$\\$\tau_D^{slow}=7.78\hspace{1em}N^{slow}=0.0$} \\
    714af & 1 &                                                                            $\tau_D=0.25\hspace{3em}N=2.8$ &                                                                           $\tau_D=0.37\hspace{3em}N=2.2$ \\
          & 2 &    \makecell{$\tau_D^{fast}=0.15\hspace{1em}N^{fast}=0.6$\\$\tau_D^{slow}=11.84\hspace{1em}N^{slow}=0.1$} &    \makecell{$\tau_D^{fast}=0.13\hspace{1em}N^{fast}=0.4$\\$\tau_D^{slow}=3.61\hspace{1em}N^{slow}=0.1$} \\
    34a6d & 1 &                                                                            $\tau_D=0.39\hspace{3em}N=3.7$ &                                                                           $\tau_D=1.62\hspace{3em}N=4.2$ \\
          & 2 &                                                            $\tau_D^{fast}=591.08\hspace{1em}N^{fast}=1.9$ &    \makecell{$\tau_D^{fast}=0.00\hspace{1em}N^{fast}=0.7$\\$\tau_D^{slow}=1.29\hspace{1em}N^{slow}=0.5$} \\
    484af & 1 &                                                                            $\tau_D=0.26\hspace{3em}N=1.9$ &                                                                           $\tau_D=0.24\hspace{3em}N=2.1$ \\
          & 2 &    \makecell{$\tau_D^{fast}=0.23\hspace{1em}N^{fast}=0.4$\\$\tau_D^{slow}=73.45\hspace{1em}N^{slow}=0.0$} &   \makecell{$\tau_D^{fast}=0.20\hspace{1em}N^{fast}=0.5$\\$\tau_D^{slow}=63.57\hspace{1em}N^{slow}=0.0$} \\
    0cd20 & 1 &                                                                            $\tau_D=0.32\hspace{3em}N=1.1$ &                                                                           $\tau_D=0.33\hspace{3em}N=1.0$ \\
          & 2 &                                                              $\tau_D^{fast}=0.32\hspace{1em}N^{fast}=0.2$ &                                                             $\tau_D^{fast}=0.33\hspace{1em}N^{fast}=0.2$ \\
    fe81d & 1 &                                                                            $\tau_D=0.27\hspace{3em}N=1.6$ &                                                                           $\tau_D=0.24\hspace{3em}N=1.7$ \\
          & 2 &   \makecell{$\tau_D^{fast}=0.25\hspace{1em}N^{fast}=0.3$\\$\tau_D^{slow}=177.40\hspace{1em}N^{slow}=0.0$} &  \makecell{$\tau_D^{fast}=0.22\hspace{1em}N^{fast}=0.4$\\$\tau_D^{slow}=133.87\hspace{1em}N^{slow}=0.0$} \\
    19e3e & 1 &                                                                            $\tau_D=0.25\hspace{3em}N=2.0$ &                                                                           $\tau_D=0.24\hspace{3em}N=2.2$ \\
          & 2 &   \makecell{$\tau_D^{fast}=0.23\hspace{1em}N^{fast}=0.4$\\$\tau_D^{slow}=146.64\hspace{1em}N^{slow}=0.0$} &   \makecell{$\tau_D^{fast}=0.20\hspace{1em}N^{fast}=0.5$\\$\tau_D^{slow}=78.29\hspace{1em}N^{slow}=0.0$} \\
    c1204 & 1 &                                                                            $\tau_D=2.69\hspace{3em}N=5.5$ &                                                                           $\tau_D=7.60\hspace{3em}N=6.1$ \\
          & 2 &  \makecell{$\tau_D^{fast}=2.35\hspace{1em}N^{fast}=3.0$\\$\tau_D^{slow}=2000.00\hspace{1em}N^{slow}=0.8$} &    \makecell{$\tau_D^{fast}=0.00\hspace{1em}N^{fast}=1.1$\\$\tau_D^{slow}=6.86\hspace{1em}N^{slow}=0.6$} \\
    \bottomrule
    \end{tabular}
    \end{table}
    
  • now I put latex in a simple template of the Overleaf online latex editor like this
    \documentclass{article}
    \usepackage{booktabs} % for \toprule and \midrule in table
    \usepackage{makecell} % for linebreaks in table cells
    
    \begin{document}
    %% here the code abvoe
    \end{document}
    
    
  • the final rendering looks like this:
  • now, we work with all_param. Here, all single correlation curves are loaded without averaging.
    def sort_fit(param_ls):
        nfcs = list(param_ls)[-1]
        triplet = list(param_ls)[-2]
        array = np.array(list(param_ls)[:-2]).reshape((2, 2))
        # sort by transit times
        array = array[:, array[0, :].argsort()]
        A_fast = float(array[1, 0])
        A_slow = float(array[1, 1])
        N_fast = A_fast * (triplet * float(nfcs))
        N_slow = A_slow * (triplet * float(nfcs))
        t_fast = float(array[0, 0])
        t_slow = float(array[0, 1])
        # for pex5: want to plot t_slow
        if np.isnan(t_slow):
            # 1-component fit
            out = t_fast, N_fast, pd.NA, pd.NA
            # out = pd.NA, pd.NA, t_fast, N_fast
        # 2-component fits
        elif f'{A_fast:.0%}' == '100%':
            out = t_fast, N_fast, pd.NA, pd.NA
            # out = pd.NA, pd.NA, t_fast, N_fast
        elif f'{A_slow:.0%}' == '100%':
            # out = t_slow, N_slow, pd.NA, pd.NA
            out = pd.NA, pd.NA, t_slow, N_slow
        else:
            out = t_fast, N_fast, t_slow, N_slow
        return out
    
    def sort_fit_legend(param_ls):
        species = param_ls[0]
        component = param_ls[1]
    
        if species == 1:
            legend = '$\\tau_D$ from\n1 species fit'
        elif (species == 2) and (component == 'fast'):
            legend = '$\\tau_D$ from\nfast sp. of 2 sp. fit'
        elif (species == 2) and (component == 'slow'):
            legend = '$\\tau_D$ from\nslow sp. of 2 sp. fit'
        return legend
    
    all_param[['t_fast', 'N_fast', 't_slow', 'N_slow']
              ]= all_param[['txy1', 'txy2', 'A1', 'A2', 'T1', 'N (FCS)']
                           ].apply(lambda x: sort_fit(x), axis=1, result_type='expand')
    
    all_param = pd.wide_to_long(all_param, stubnames=['t', 'N'],
                                i=['name_of_plot', 'Diff_species', 'processing'],
                                j='fit component',
                                sep='_', suffix=r'\w+')
    
    all_param = all_param.reset_index()
    # if Diff_species is 1, there is only 1 component
    all_param = all_param[~((all_param['fit component'] == 'slow') & (all_param['Diff_species'] == 1))]
    all_param = all_param.reset_index()
    
    all_param['legend'] = all_param[['Diff_species', 'fit component']].apply(
        lambda x: sort_fit_legend(x), axis=1)
    print('before dropping NaNs')
    print('1 species fit: {}'.format(len(all_param[all_param["legend"] == "$\\tau_D$ from\n1 species fit"])))
    print('slow sp of 2 sp fit: {}'.format(len(all_param[all_param["legend"] == "$\\tau_D$ from\nslow sp. of 2 sp. fit"])))
    print('fast sp of 2 sp fit: {}'.format(len(all_param[all_param["legend"] == "$\\tau_D$ from\nfast sp. of 2 sp. fit"])))
    
    all_param = all_param[~pd.isna(all_param['t'])]
    print('after dropping NaNs')
    print('1 species fit: {}'.format(len(all_param[all_param["legend"] == "$\\tau_D$ from\n1 species fit"])))
    print('slow sp of 2 sp fit: {}'.format(len(all_param[all_param["legend"] == "$\\tau_D$ from\nslow sp. of 2 sp. fit"])))
    print('fast sp of 2 sp fit: {}'.format(len(all_param[all_param["legend"] == "$\\tau_D$ from\nfast sp. of 2 sp. fit"])))
    
    all_param = all_param[['legend', 't', 'N', 'artifact', 'processing']]
    all_param.loc[:, ['t', 'N']] = all_param.loc[:, ['t', 'N']].apply(pd.to_numeric)
    all_param
    N_param = all_param.pivot_table(values='N',
                                    index=all_param.index,
                                    columns=['processing', 'artifact', 'legend'],
                                    sort=False)
    t_param = all_param.pivot_table(values='t',
                                    index=all_param.index,
                                    columns=['processing', 'artifact', 'legend'])
    print('Statistics of fitted particle numbers')
    with pd.option_context('display.float_format', '{:.3f}'.format):
        display(pd.concat([pd.Series(N_param.median(axis=0), name='median'), N_param.describe().T], axis=1))
    print('Statistics of fitted transit times')
    display(pd.concat([pd.Series(t_param.median(axis=0), name='median'), t_param.describe().T], axis=1))
    
    before dropping NaNs
    1 species fit: 5000
    slow sp of 2 sp fit: 5000
    fast sp of 2 sp fit: 5000
    after dropping NaNs
    1 species fit: 5000
    slow sp of 2 sp fit: 4588
    fast sp of 2 sp fit: 4940
    Statistics of fitted particle numbers
    
          median count mean std min 25% 50% 75% max
    processing artifact legend                  
    0cd20 Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.266 250.000 0.253 0.068 0.094 0.193 0.266 0.303 0.489
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.043 249.000 0.056 0.047 0.000 0.043 0.043 0.043 0.264
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.029 226.000 0.036 0.024 0.000 0.029 0.029 0.029 0.177
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.208 250.000 0.210 0.033 0.132 0.186 0.208 0.229 0.298
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.193 244.000 0.175 0.074 0.000 0.176 0.193 0.212 0.319
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.014 79.000 0.049 0.068 0.000 0.003 0.014 0.060 0.239
    19e3e Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.422 250.000 0.534 0.313 0.000 0.367 0.422 0.532 1.748
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.047 250.000 0.063 0.115 0.000 0.047 0.047 0.047 0.888
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.031 250.000 0.033 0.027 0.000 0.031 0.031 0.031 0.324
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.659 250.000 0.653 0.239 0.000 0.497 0.659 0.796 1.372
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.032 249.000 0.086 0.132 0.000 0.000 0.032 0.108 0.744
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.010 250.000 0.039 0.080 0.000 0.000 0.010 0.036 0.770
    34766 Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 1.416 250.000 525.558 8274.061 0.000 1.015 1.416 2.362 130826.616
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.156 247.000 302.119 4744.397 0.000 0.156 0.156 0.156 74564.294
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.068 249.000 0.515 6.186 0.000 0.068 0.068 0.068 97.572
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 2.158 250.000 2.526 0.922 0.176 1.848 2.158 3.524 4.153
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.000 248.000 0.160 0.365 0.000 0.000 0.000 0.056 1.599
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.000 250.000 0.225 0.691 0.000 0.000 0.000 0.007 4.143
    34a6d Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 1.377 250.000 1.954 1.534 0.000 1.117 1.377 2.135 7.288
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.000 237.000 0.261 0.620 0.000 0.000 0.000 0.381 7.287
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.000 249.000 0.381 1.252 0.000 0.000 0.000 0.054 6.970
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 3.059 250.000 2.908 0.666 0.000 2.526 3.059 3.322 4.297
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.000 250.000 0.019 0.157 0.000 0.000 0.000 0.000 1.549
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.000 250.000 0.007 0.062 0.000 0.000 0.000 0.000 0.819
    484af Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.413 250.000 0.508 0.251 0.000 0.371 0.413 0.521 1.576
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.343 250.000 0.390 0.202 0.000 0.293 0.343 0.406 1.301
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.014 247.000 0.021 0.020 0.000 0.010 0.014 0.024 0.140
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.588 250.000 0.601 0.213 0.000 0.472 0.588 0.726 1.269
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.074 250.000 0.079 0.060 0.000 0.074 0.074 0.074 0.656
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.053 250.000 0.052 0.016 0.000 0.053 0.053 0.053 0.225
    714af Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.767 250.000 1.131 0.801 0.000 0.618 0.767 1.472 3.447
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.434 250.000 0.397 0.315 0.000 0.225 0.434 0.551 2.821
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.023 247.000 0.059 0.169 0.000 0.010 0.023 0.039 1.567
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.997 250.000 1.003 0.419 0.229 0.633 0.997 1.262 2.225
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.056 250.000 0.169 0.208 0.000 0.000 0.056 0.307 0.812
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.011 250.000 0.034 0.056 0.000 0.000 0.011 0.050 0.382
    No correction Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.268 250.000 0.257 0.071 0.118 0.194 0.268 0.309 0.508
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.253 249.000 0.235 0.087 0.000 0.173 0.253 0.297 0.505
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.003 156.000 0.012 0.034 0.000 0.002 0.003 0.005 0.259
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.189 250.000 0.188 0.041 0.012 0.169 0.189 0.212 0.292
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.166 249.000 0.155 0.059 0.000 0.137 0.166 0.195 0.275
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.006 226.000 0.008 0.008 0.000 0.003 0.006 0.011 0.046
    c1204 Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 3.582 250.000 1800.956 16542.991 0.000 2.368 3.582 4.792 186293.638
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.252 248.000 0.366 0.804 0.000 0.252 0.252 0.252 8.390
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.062 248.000 0.190 0.663 0.000 0.062 0.062 0.062 6.570
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 4.392 250.000 5.219 2.156 0.000 3.632 4.392 7.214 11.235
        \(\tau_D\) from\nfast sp. of 2 sp. fit 2.757 242.000 3.311 2.830 0.000 1.054 2.757 4.820 11.086
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.629 177.000 1.297 1.914 0.000 0.000 0.629 1.264 8.789
    fe81d Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.326 250.000 0.393 0.197 0.000 0.298 0.326 0.377 1.327
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.030 250.000 0.044 0.100 0.000 0.030 0.030 0.030 1.038
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.018 250.000 0.022 0.022 0.000 0.018 0.018 0.018 0.235
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.427 250.000 0.440 0.157 0.000 0.336 0.427 0.543 0.900
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.062 228.000 0.154 0.181 0.000 0.009 0.062 0.248 0.790
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.022 235.000 0.082 0.133 0.000 0.005 0.022 0.074 0.545
    ff67b Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.502 250.000 0.557 0.202 0.185 0.428 0.502 0.603 1.429
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.029 250.000 0.051 0.121 0.000 0.029 0.029 0.029 0.978
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.014 250.000 0.020 0.026 0.000 0.014 0.014 0.014 0.227
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.412 250.000 0.438 0.120 0.236 0.358 0.412 0.489 0.837
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.337 250.000 0.345 0.127 0.000 0.279 0.337 0.399 0.781
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.015 249.000 0.019 0.030 0.000 0.011 0.015 0.020 0.422
    Statistics of fitted transit times
    
          median count mean std min 25% 50% 75% max
    processing artifact legend                  
    0cd20 Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.323617 250.0 0.320864 0.020821 0.249462 0.308254 0.323617 0.335541 0.359334
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.000172 249.0 0.060898 0.119981 0.000100 0.000100 0.000172 0.004964 0.333456
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.299218 226.0 0.652991 5.386862 0.230471 0.299212 0.299218 0.299225 81.276695
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.324881 250.0 0.326603 0.027692 0.259626 0.309170 0.324881 0.343199 0.429618
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.330889 244.0 0.296751 0.115240 0.000100 0.304649 0.330889 0.336307 0.565352
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.344133 79.0 96.592373 358.830694 0.232596 0.291604 0.344133 3.224768 1999.956477
    19e3e Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.248882 250.0 0.246302 0.043006 0.034228 0.231552 0.248882 0.266995 0.492987
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.000100 250.0 0.007082 0.035425 0.000100 0.000100 0.000100 0.000100 0.311329
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.231668 250.0 11.096372 128.807510 0.160041 0.231668 0.231668 0.231668 1999.965381
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.237834 250.0 0.241093 0.056889 0.123511 0.205615 0.237834 0.265857 0.711353
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.005744 249.0 0.046721 0.065354 0.000100 0.000100 0.005744 0.105139 0.312681
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.214517 250.0 13.995457 40.382504 0.130520 0.184269 0.214517 2.095036 392.234249
    34766 Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.282741 250.0 1.960813 10.421541 0.000109 0.211177 0.282741 0.804156 161.798448
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.000100 247.0 0.140035 1.065767 0.000100 0.000100 0.000100 0.000100 12.772683
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.199918 249.0 40.935269 281.035522 0.036570 0.199918 0.199918 0.199918 2000.000000
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.691528 250.0 2.826474 3.458340 0.150428 0.417778 0.691528 5.433018 19.065626
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.051069 248.0 0.055039 0.052744 0.000100 0.020852 0.051069 0.065619 0.366803
        \(\tau_D\) from\nslow sp. of 2 sp. fit 7.512514 250.0 8.054456 4.969898 0.134926 5.006964 7.512514 10.973836 30.326475
    34a6d Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.284963 250.0 2.814169 7.351230 0.001338 0.236236 0.284963 0.512840 40.170929
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.063051 237.0 0.220596 2.374881 0.000100 0.045802 0.063051 0.087420 36.620450
        \(\tau_D\) from\nslow sp. of 2 sp. fit 9.022691 249.0 10.867661 7.960415 0.106761 6.005959 9.022691 13.070269 47.255466
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 1.834290 250.0 2.268876 1.877872 0.088689 0.856312 1.834290 3.069606 10.252194
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.052587 250.0 0.052591 0.005518 0.016660 0.052587 0.052587 0.052587 0.122504
        \(\tau_D\) from\nslow sp. of 2 sp. fit 7.066470 250.0 7.212279 1.224819 3.041586 7.066470 7.066470 7.066470 20.280015
    484af Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.265457 250.0 0.260147 0.040656 0.128895 0.240648 0.265457 0.283187 0.444097
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.233343 250.0 0.222758 0.050222 0.000100 0.200308 0.233343 0.257370 0.328613
        \(\tau_D\) from\nslow sp. of 2 sp. fit 69.916879 247.0 133.744606 200.200824 0.142299 38.940604 69.916879 147.092548 1924.273303
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.241658 250.0 0.243606 0.044011 0.113022 0.213493 0.241658 0.271191 0.435722
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.000100 250.0 0.004805 0.031999 0.000100 0.000100 0.000100 0.000100 0.357377
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.221663 250.0 0.666995 3.190823 0.155482 0.221663 0.221663 0.221663 27.635426
    714af Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.233111 250.0 0.411415 0.475560 0.027502 0.204674 0.233111 0.326329 3.026657
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.151704 250.0 0.139625 0.181602 0.000100 0.073383 0.151704 0.176694 2.807864
        \(\tau_D\) from\nslow sp. of 2 sp. fit 15.624278 247.0 33.247501 63.227049 0.137890 6.196147 15.624278 35.200767 651.235702
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.366270 250.0 0.401344 0.128259 0.187158 0.329494 0.366270 0.431351 1.122314
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.077699 250.0 0.101588 0.078889 0.000100 0.050206 0.077699 0.153557 0.330826
        \(\tau_D\) from\nslow sp. of 2 sp. fit 3.390052 250.0 4.456640 4.540913 0.198274 1.685021 3.390052 5.868710 36.023611
    No correction Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.356176 250.0 0.356892 0.017251 0.316204 0.345597 0.356176 0.367787 0.422647
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.348573 249.0 0.330113 0.080803 0.000100 0.334796 0.348573 0.359479 0.422648
        \(\tau_D\) from\nslow sp. of 2 sp. fit 42.013383 156.0 234.082459 382.506491 0.301974 10.840074 42.013383 273.555579 1999.121327
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.437270 250.0 0.507154 0.229794 0.351314 0.403647 0.437270 0.498338 2.429498
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.391268 249.0 0.389270 0.136774 0.000100 0.365634 0.391268 0.428259 1.044106
        \(\tau_D\) from\nslow sp. of 2 sp. fit 15.194065 226.0 89.237400 267.388024 0.352227 5.441904 15.194065 43.653002 2000.000000
    c1204 Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.651378 250.0 4.339958 9.135442 0.000100 0.255066 0.651378 3.500988 70.322682
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.000100 248.0 0.194788 1.877687 0.000100 0.000100 0.000100 0.000100 26.384233
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.195006 248.0 1.556390 12.510745 0.099500 0.195006 0.195006 0.195006 193.005430
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 4.027498 250.0 39.287596 137.224405 0.087313 2.197479 4.027498 42.922206 1999.613089
        \(\tau_D\) from\nfast sp. of 2 sp. fit 3.121523 242.0 24.498094 44.439024 0.000100 0.078008 3.121523 35.018711 300.809427
        \(\tau_D\) from\nslow sp. of 2 sp. fit 1182.632990 177.0 1021.164847 960.486419 0.148180 10.943002 1182.632990 1999.979383 2000.000000
    fe81d Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.270886 250.0 0.264426 0.040805 0.126198 0.239281 0.270886 0.292043 0.417882
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.000100 250.0 0.005696 0.031403 0.000100 0.000100 0.000100 0.000225 0.277912
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.253200 250.0 1.574326 8.690581 0.159645 0.253200 0.253200 0.253200 91.512999
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.237785 250.0 0.239106 0.035075 0.162728 0.215458 0.237785 0.258728 0.452019
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.095902 228.0 0.111506 0.108528 0.000100 0.000513 0.095902 0.223517 0.452037
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.238129 235.0 25.725146 77.901884 0.148100 0.201307 0.238129 7.581677 739.642771
    ff67b Hs-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.230441 250.0 0.239749 0.047058 0.144283 0.213528 0.230441 0.250664 0.577085
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.000100 250.0 0.007750 0.041582 0.000100 0.000100 0.000100 0.000100 0.288617
        \(\tau_D\) from\nslow sp. of 2 sp. fit 0.203118 250.0 5.851806 61.517373 0.148434 0.203118 0.203118 0.203118 922.471752
      Tb-PEX5-eGFP \(\tau_D\) from\n1 species fit 0.280494 250.0 0.281235 0.040631 0.184676 0.253043 0.280494 0.306884 0.405158
        \(\tau_D\) from\nfast sp. of 2 sp. fit 0.237870 250.0 0.229680 0.058269 0.000100 0.206773 0.237870 0.265449 0.362244
        \(\tau_D\) from\nslow sp. of 2 sp. fit 31.791098 249.0 46.205757 52.410003 0.158099 14.756188 31.791098 58.344302 455.148873
  • now we plot fit outcomes (transit times, particle numbers) after neural network prediction and cut and stitch correction. For this PEX5 data, we compare Hs-PEX5-eGFP (no artifacts) against Tb-PEX5-eGFP (peak artifacts). The gold standard was the 1 component fit of Hs-PEX5-eGFP without any correction (see median values plotted with a horizontal line). Here, the results were much more mixed, so I kept the 1 species fit, which worked best, and all 2 species fits. The best method was 0cd20 (only 1 species fits) which corrected the Tb-PEX5-eGFP traces without distorting the Hs-PEX5-eGFP traces. All other models had problems.
    pub_param = all_param.replace(['0cd20'], '0cd20: large model\n(200 MB), 6 levels,\npool size=4, scaler\n=quantile transform\n(Gaussian pdf)')
    pub_param = pub_param.replace(['34a6d'], '34a6d: small model\n(7 MB), 3 levels,\npool size=4, scaler=l2')
    pub_param = pub_param.replace(['484af'], '484af: large model\n(275 MB), 7 levels,\npool_size=2, scaler\n=standard')
    pub_param = pub_param.replace(['fe81d'], 'fe81d: large model\n(186 MB), 4 levels,\npool_size=4, scaler\n=standard')
    pub_param = pub_param.replace(['ff67b'], 'ff67b: small model\n(14 MB), 5 levels,\npool_size=4, scaler\n=minmax')
    pub_param = pub_param.replace(['19e3e'], '19e3e: large model\n(172 MB), 3 levels,\npool_size=4, scaler\n=standard')
    pub_param = pub_param.replace(['34766'], '34766: middle-sized\nmodel (73 MB), 5 levels,\npool_size=4, scaler\n=robust')
    pub_param = pub_param.replace(['c1204'], 'c1204: large model\n(312 MB), 9 levels,\npool_size=2, scaler\n=robust')
    pub_param = pub_param.replace(['714af'], '714af: large model\n(234 MB), 5 levels,\npool_size=4, scaler\n=maxabs')
    # to make log plot work with a catplot violin plot, we have to do the
    # log transform manually
    pub_param.loc[:, 't'] = pub_param.loc[:, 't'].apply(lambda x: np.log10(x))
    
    g = sns.catplot(data=pub_param,
                    y='t',
                    x='processing',
                    row='artifact',
                    hue='legend',
                    height=9,
                    aspect=3.5,
                    legend_out=True,
                    kind='violin',
                    sharey=True,
                    showfliers=False,
                    scale='width',
                    cut=0)
    g._legend.remove()
    for i, ax in enumerate(g.axes):
        clean = pub_param[(pub_param['processing'] == 'No correction') &
                          (pub_param['artifact'] == 'Hs-PEX5-eGFP')]
        median = clean['t'].median()
        median_text = 10**median
        line = ax[0].axhline(median, lw=4, label='', ls='--')
        line_legend = {f'\n$\\tau_{{exp}}={median_text:.2f}ms$' : line}
        g._legend_data.update(line_legend)
    g.add_legend(g._legend_data)
    
    g.fig.suptitle('Model performance in simulation → prediction → correction pipeline successfully (transit times)', size=25)
    
    plt.setp(g.axes, yscale='linear', ylabel=r'log transit time $\tau_{D}$ $[ms]$',
             xlabel='', xlim=None)
    for i, ax in enumerate(g.axes):
        ylab = ax[0].get_yticklabels()
        # because seaborns violinplot does not support kde calculation in log values,
        # I have to do this manually, by first log-transforming the data, now
        # extracting the yticklabels and manually transforming them back.
        ylab_power = [10**lab.get_position()[1] for lab in ylab]
        ax[0].set_yticklabels(ylab_power)
    # the following code snippet makes minor grid lines in log plots visible. Since I have to
    # hack the log axis here to make violin catplots work, this is not possible
    # for ax in g.axes.flatten():
    #     ax.grid(visible=True, which='both', axis='y')
    g.tight_layout()
    savefig = f'./data/exp-220227-unet/jupyter/analysis3_pex5_transit-times'
    plt.savefig(f'{savefig}.pdf', dpi=300)
    # os.system(f'pdf2svg {savefig}.pdf {savefig}.svg')
    plt.close('all')
    
    g = sns.catplot(data=pub_param,
                    y='N',
                    x='processing',
                    row='artifact',
                    hue='legend',
                    height=9,
                    aspect=3.5,
                    legend_out=True,
                    kind='violin',
                    sharey=True,
                    showfliers=False,
                    scale='width',
                    cut=0)
    g._legend.remove()
    for i, ax in enumerate(g.axes):
        clean = pub_param[(pub_param['processing'] == 'No correction') &
                          (pub_param['artifact'] == 'Hs-PEX5-eGFP')]
        median = clean['N'].median()
        line = ax[0].axhline(median, lw=4, label='', ls=':')
        line_legend = {f'\n$N{{exp}}={median:.2f}$' : line}
        g._legend_data.update(line_legend)
    g.add_legend(g._legend_data)
    
    g.fig.suptitle('Model performance in Simulation → prediction → correction pipeline (particle numbers)', size=25)
    
    plt.setp(g.axes, yscale='linear', ylabel=r'particle number $N$ $[fl^{-1}]$',
             xlabel='', ylim=[-0.1, 3])
    # for i, ax in enumerate(g.axes):
    #     ylab = ax[0].get_yticklabels()
    #     # because seaborns violinplot does not support kde calculation in log values,
    #     # I have to do this manually, by first log-transforming the data, now
    #     # extracting the yticklabels and manually transforming them back.
    #     ylab_power = [10**lab.get_position()[1] for lab in ylab]
    #     ax[0].set_yticklabels(ylab_power)
    
    g.tight_layout()
    savefig =  f'./data/exp-220227-unet/jupyter/analysis3_pex5_particle-numbers'
    plt.savefig(f'{savefig}.pdf', dpi=300)
    # os.system(f'pdf2svg {savefig}.pdf {savefig}.svg')
    plt.close('all')
    
    /tmp/ipykernel_371129/654097065.py:48: UserWarning: FixedFormatter should only be used together with FixedLocator
      ax[0].set_yticklabels(ylab_power)
    
  • this is the two plots as PDFs, first the comparison of transit times, second the comparison of particle numbers

2.6.14 final remarks

  • This is a table for quick lookup of all metrics of the best models after 100th epoch with hparams
    run valauc valf1 0.5 valprec 0.5 valrecall 0.5 model size hpbatchsize hpfirstfilters hpinputsize hplrpower hplrstart hpnlevels hppoolsize hpscaler
    484af471c61943fa90e5f78e78a229f0 0.9814 0.9187 0.9091 0.9285 275 MB 26 44 16384 (here: 14000) 1 0.0136170138242663 7 2 standard
    0cd2023eeaf745aca0d3e8ad5e1fc653 0.9818 0.9069 0.8955 0.9185 200 MB 15 23 16384 (here: 14000) 7 0.0305060808685107 6 4 quantg
    fe81d71c52404ed790b3a32051258da9 0.9849 0.9260 0.9184 0.9338 186 MB 20 78 16384 (here: 14000) 4 0.0584071108418767 4 4 standard
    ff67be0b68e540a9a29a36a2d0c7a5be + 0.9859 0.9298 0.9230 0.9367 14 MB 28 6 16384 (here: 14000) 1 0.0553313915596308 5 4 minmax
    19e3e786e1bc4e2b93856f5dc9de8216 0.9595 0.8911 0.8983 0.8839 172 MB 20 128 16384 (here: 14000) 1 0.043549707353273 3 4 standard
    347669d050f344ad9fb9e480c814f727 + 0.9848 0.9246 0.9254 0.9238 73 MB 10 16 8192 (here: 14000) 1 0.0627676336651573 5 4 robust
    c1204e3a8a1e4c40a35b5b7b1922d1ce 0.9858 0.9207 0.9179 0.9234 312 MB 14 16 16384 (here: 14000) 5 0.0192390310290551 9 2 robust
    714af8cd12c1441eac4ca980e8c20070 + 0.9843 0.9304 0.9257 0.9352 234 MB 9 64 4096 (here: 14000) 1 0.0100697459464075 5 4 maxabs
    34a6d207ac594035b1009c330fb67a65 + 0.9652 0.8613 0.8598 0.8629 7 MB 17 16 16384 (here: 14000) 5 0.0101590069352232 3 4 l2
  • I did further evaluation of the most promising model, 0cd20, and applied averaging and set to zero corrections. For these results, see exp-220316-publication1